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2k October 1973 


MEMORANDUM FOR: Members of the Curriculum Committee 

SUBJECT: Agenda for Meeting, Thursday, 25 October 


1. The Curriculum Committee will meet on Thursday, 

25 October; you will be notified of the time and place by 
telephone. 

2. The agenda includes: 

a. Course reports, critiques, and schedule 
formats; 

b. The Self-Study Program; 

c. Requirements and production of TV tapes. 

3. Attached as background information are excerpts 
from previous Curriculum Council minutes dealing with Item (a) ; 

a paper on the Self-Study Program is included for your information. 


STATINTL 
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31 October 1973 


MEMORANDUM FOR: Members of the Curriculum Committee 

SUBJECT: Minutes of Meeting, 25 October 1973 


STATINTL 

STATINTL 


STATINTL 

STATINTL 


STATINTL 

STATINTL 


1. The Curriculum Committee met in the DTR's Conference Room 
at 1400 hour s. 25 October 1973- All members vere present with the 

exception of | 

request. 


STATINTL 


2 . Beginning with a discussion of the Self- Study Prog ram, 

directed the members' attention to I "I s memo STATINTL 


on the same subject dated 24 October 1973. The memo outlines the 
options open now in the Self-Study Program, reminding that the 
words "self-study" mean different things to different people. Mr. 

added that the DTR has expressed the need for the use of 


video cassette tapes as a primary teaching medium in management 
courses, and the Director hopes this medium will also be applied 
to other OTR courses. This would still require the time of an 
instructor who would conduct the course or handle a seminar, 
discussions, and the problems presented; so therefore, in the 
most precise definition, it would not really be a self-study 
activity. Nevertheless, a requirement has been levied for the 
development of cassettes and the acquisition of equipment . Mr. 

I I said that of the video cassettes available today, those 

helpful to us exist only in the fields of management and computer 
science; and while there are some available in specific skills 
training, no t many are relevant to the needs of this Agency. 

said that we have often purchased "packaged things," 


e.g., the Grid and the former AMP (which drew much criticism). 

Before we purchase a package of management theory in cassette form, 
we should look at its relevance to the managerial practices of 
this Agency. If it isn't related, we should not put a large invest- 
ment in that kind of program. Given the shortage of funds ($6,000) 
for this activity and given the fact that we are still in the 
experimental stage, the Committee wants to learn the essential 
steps first before starting into courses. After h is attenda nce 
at an Audio Visual Seminar in Chicago in November, 


be able to provide more information of a technical and financial 
na ture. In the mea ntime, we will proceed with options (c) and (d) 


may STATINTL 


of 


paper . 


2 . 


is taping training units; 


their goal is to reach the point where the tapes would be self- 
sustaining. The project is far from complete, but it has wide 
applicability. The project will have to be completed at a time 
when we are "fatter" personnel-wise and money-wise, for its 
successful completion would require the full-time concentration of 
several instructors. 
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critiques, but he felt that they should not be permanently attached 
to the End-of-Course report which goes to the DTR— with the exception 
being course critiques for new courses* He suggested the following 
routing for the End-of-Course report: instructor to Branch or Unit 
Chief; SA/OT (if the reports contain items of interest to him); 
C/PDS/DTG (critiques attached; to be reviewed and returned to Unit 
Chief for retention); C/PDS; DTR. After discussion, the Committee 
decided 

(a) Critiques would be required in all OTR courses; 

(b) Critiques will not be forwarded to the DTR as part of 
the End-of-Course report; exception: new courses; 

(c) Signatures on the critiques are not required; 

(d) Critiques and End-of-Course reports should include a 
_question/paragraph on whether or not objectives were 
met. 


The Committee agreed that a policy paper should be devised on 
critiques and End-of-Course Reports; hopefully, the paper will 
show a list of things to be covered in critiques, but will allow 
flexibility in format. On End-of-Course Reports, the Committee 
began a discussion of things to be included. Information is needed 
on EEO statistics— should they be included in Reports? The 
breakdown of the class by Directorates is helpful. 


9. The 
7-8 November 


meeting ended with a discussion of the agen da for the 
meeting I 


STATINTL 


STATINTL 


-3- 
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25X1A 

25X1A 
25X1 A 

25X1 A 
25X1 A 

0 

25X1A 


25X1 A 



changes. ) The first running of the MEDC under SIWA auspices will be in 
“the* fall. 'There will be no changes in the Course until the winter. ^Mr.' 
will draw up a reorganization plan of his faculty. 




It was also noted that Mr. Cunningham is quite concerned about 
"tribalism" in the Agency and is of the opinion that the MEDC is a good 
vehicle to break down the barriers and might not be too amenable to sug- 
gested changes in the Course. 


c 


] told the Council that £ 


] is attempting to25X1A 


identify midcareerists..’ Q 


]was advised by the DDTR that OTR should 


not become involved in CS career planning and that each Directorate should 
be responsible for its own planning. 


Friday, 16 July 1971: 




reading what is now required 


3. The ses sion began with | 
in End-of- Course Reports. There seemed to be little on the content of the 
reports. Most of the discus sion centered on whether the reports should 

explained that the middleman 


go through a "middleman. 


was used primarily to highlight the important developments and -to pinpoint 
those things that would require action. The Council was unanimous in the 
opinion that course coordinators and chiefs of schools were in a much more 
advantageous position to flag problems and highlights. It was agreed that 
the End-of- Course Reports will go directly to DDTR, by-passing the 


“middleman. M 




[ 


] outlined on the blackboard the objectives of 


course reports: 


s 

i-4 

t.y. 


1. 

2 . 


To inform School Chiefs and DTR 
To report on that running 

a. Quality 

b. Student reaction 

c. Changes introduced' 

d. Problems 


As further guidelines to aid in preparation of the reports, Mr. 

]] informed the Council that DTR is not concerned with the physi- 
cal aspects of a course running but he is interested in what changes have 
’ t new exercises have been inaug ura ted and wr ~ J ~*"' 1 — * 


been madc^ 

"might have done an outstanding job. 


student 


mu 


Curriculum Council 
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[will prepare?^ -m 


^. er !.. of . thc : Co ^ ricil ..^' t ii n 8 . fo L tl \ & l ^ idelil l c s 

The guidelines,: when approved, will be distributed to aU^schoor’cK're^s"" 
over the signature of the DTIt, 


I 

N 

4. The subject of student critiques was next on the agenda. The con- 

census of the Council was thaF" student critiques really were of little value* 
(Except for nev/j:our seZjtwas al so^ag^ would^not be po'sTible " 

.™ run a cour se. without Jjayitig student critiques.^" It was 'recognized 'that* 
each school has different problems on student critiques but all members 
were in agreement that the student should address his critiques ‘to'method 
rather than personality. Council members wished to make further studies 

..before coming to any conclusions. | j asked DDTR for guidance. 

' teS^^ Med.jthat,^ "We reallvjp an’ t^giye guidance. I think that you should 
feel free to experiment — ..even tothe point o'f dropping it.' " "DDTR told* ' 
the Council that he would discuss this prdblem’'with DTR^' Further dis- 
c us sion on student critiques will take place in the fall. 

5. 


» i _ _ • \ 

Next meeting will take place in September, 

!l - 

\ 


II 


m 


Date to be set later. 




( 

f 

ik 

x/4 
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I 19 October 1971 1300 


The subject of student critiques discussed. It was generally agreed it 
did not seem appropriate to critique the same course over and over again 
because nothing really resulted from this type of exercise. Critiques are 
much more valuable from those courses directed at the mid-level to senior 

that with25^ 


It was suggested by 


officer, e. g. , AIS, COS .Seminar, 
the exception of new courses, or where changes have been made, ciitiqucs 
should be at the option of the school chief. When critiques, are called for, 
only forward those that have something to say. If course reports arc not 
satisfactory, front office can call. In conjunction with this discussion, an 


additional way to get good fe edback is by m onitoring individual presentations. 


Chairman recommended that 


spot check the courses. He will 


talk to school chiefs, not the front office. 


The following recommendations were spelled out; 


1. 'Cri%-iq;tfes voluntary -- school chiefs to decide when to 
forward. School can file critiques and invite DTR to see them. 


2. Critiques not necessary, in long established courses. 


) 3. Critiques should not be used in those courses in which 
student not capsble of making constructive criticism. 


4. Critiques will be used in new courses. 

5. Critiques will be used whe.p^contemplating change. 


6. In lieu of critiques, schbol chiefs will provide the DTR with 
better course reports. — ... 


19 October 1971 


brought up the subject of End-of- Course reports. He men- 
tioned that he had thought that the "middle-man" concept had been discarded. 


make his 


He stated that he did not think it necessary that 

comments on the routing sheet inasmuch as he is talking to the DTR; not the 

] stated that he would now make his notes on 


course chief. E 


separate work sheet and at appropriate time submit them to the front office. 


Curriculum Council 
19 October 1971 
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Agenda 

Curriculum Council . 25X1 




3. 1 I EOD • 25X1A 

a. Should any course be mandatory, • 

b. If mandatory -- any exceptions, . 

c. Who can exempt, 

d. Time limit, 

e. Test as a substitute. 



Student Critiques - this subject was discussed at the CC meeting of 
16 July 71. Council members wished to have more time to consider 
the problem. . 

a. Value of critiques, 

b. Should critiques be invited, 

c. Should all courses have student critiques, 

d. Can we devise one format for all courses. 


e. What do you look for? 

fMj 
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Student Evaluations(^_^ - — 

a. Of what value are student evaluations and to whom, 

b. Should evaluations be dropped completely, 

c. Identify courses in which students should be evaluated, 

d. What should determine whether students are evaluated, e. g. , 
length of course, specialized course. 


6. Admittance procedures to OTR courses 

a. Where does the authority lie - to deny admittance, 

b. Should prerequisite for entrance into courses be strictly 
adherred to - at what point does course chairman or school 
chief exercise his authority - if such exists, . 

c. Should the authorities prerogatives be spelled out and included 
in course descriptions? 



7. Role of the Training Officer 

a. Most training officers function only on a part-time basis as far 
as training is concerned - how this effected communication 
between OTR and components, 

b. Should Divisional training officer be ST Careerist, 

c. Where does responsibility of OTR training officer lie in main- 
taining liaison with not only divisional but component training 
officer, 

d. Establishment of a senior officer as OTR training officer. 


8. What is the minimum capability OTR is expected to maintain in 
special operations (PM) 
a* People 
b. Skills 
c« Courses 

25X1 


9. Resolution for the responsibility for Overseas Orientation 

a. OS has surveyed the problem - what are the results, 

b. . Does OTR intend to keep it - if so where does it belong - OS, SIWA. j 

10. Possible input on understanding of foreign cultures into courses other 
than IWA . •, 

a. What are we doing now, • 

b. What further can be done, ' . , 

c. Case histories, j 

d. Should Language School have a function. 

HOT PfHY 

Approved For Release 2002H 1/01 : CIA-RDP78-06215A0(^^ < 1^)^ 6 §° U 5^ Agcnda 

SECRET 





r 


j - ; 

} d 

* * i. i t. :J ** 

Approved For Release 2002/11/01 : CIA-RDP78-06215A08#*0001 0007-0 


Curriculum Council Meeting 
12 November 1971 


25X1 A 


In attendance: 


Mr. Cunningham, DTR 


p, Chairman 


beretary 


1. This meeting was called to explain to Mr. Cunningham certain 
recommendations that have been made to him by the Curriculum Council. 
The Chairman stated that the recommendations are certainly not firm and 
the collective minds of the CC are open and seeking answers to the 
recommendations. The Chairman in conjunction with the members of 
the CC decided to discuss three items that were considered at the 18-20 
October meeting of the Curriculum Council: Student Critiques, the 
Catalog, and the proposed new course - Agency Trends & Highlights. 

The first two items were discussed, the other will be considered at a 
later date. 




25X1 A 


The subject of "Student Critiques" was first on the agenda. As 
background information the Chairman opened this discussion by making 
the general statement that there is a feeling among the School Chiefs 
that there is more time given by the front office to student critiques than 
to the end-of-course reports. The chairman gave as further background 
^information the recommendations that were set forth by the CC at the 
18-20 October 1971 meeting (see para 4 of the minutes of that meeting). 
"The Chairman suggested that certainly we should not discontinue the 
critiques but that it is incumbent upon the school chiefs to use these 
critiques to add to the knowledge and to get a better feel for a particu- 
lar course. 


| suggested that it should be mandatory that we insert a 
paragraph in the End-of-Course report on student critiques. He also 
suggested that student critiques can be informal and not demand that 
they be written. He suggested that critiques are an immediate manage- 
ment tool. lie cited as an example the use of Course Coordinator time 
during the BOC which is held every Friday. 
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j. 


# ML» _punningham suggested that the School Chiefs look into constant 
o ^t^criodic critiques such as is done in the Senior Sem inar. 

He state d keep such critiques flexible but; that it is o ne way to attain a 
.sense of how we are doing during the course , ~ 


raised the point of whether t he instructors a re made 


stated 


aware of critical comments made about them. { f atcllcu 

that this is a difficult problem stating that some instructors are better 
performers than others and at times there are invidious sets of 
distinctions and that there have been no real specific results from the 
critiques. The same instructors are praised course after course and 
others are panned course after course and those who have been panned 
have come to the conclusion that they will never improve. Mr. Cunning- 
ham suggested that for those instructors panned course after course that 
it might be better that he do something else. DDTR stated that this can 
become a most difficult problem and we have at times tried to place these 
p e ople in other positions without too much success. 


(£j 

7 ) 


I pointed out that in his studying of critiques that in 
some instances students have commented on certain elements of a course, 
e. g. , not enough time to read the reading material, a given speaker has 
bombed or a lecture overdrawn and no comment on the part of 'the instruc 
tor as to whether these comments have been noted. 

DDTR noted that course critiques give independent points of view and 
counterbalance the fact that some instructors do not come to grips with 
problems. 

Ml »- Cunningham remarked that the content is most jjrfpor tan t arm 
that the critiques should be substantive -- what e.ndriTowC course should 
be given, ' ' 

Critiques have been valuable in that a number of changes have been 
brought about by student critiques. Mr. Cunningham remarked that he 
does not_ pay more attention to critiques than End-of- Course reports. 

If an End-of-Course Report, is 'good, e. g. „ AOC, it is really not necessary 
t0 f ead t^fifiticiu, dh the otherhand ^f a ha lf a doz en or so shTd^ tT^ kT 

- to a K AC f . In long establis hed courses, it is* 
?. 0t for c ritique s one then 

linSa. Mr. Cunningham stated that being allowed to submit critiques gives 
the student a sense of participation, which is needed. Mr. Cunningham 

I 

Curriculum Council 
'•-''pr 12 November 1971 
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s ugg e s ted that the c rit iques should vary with different courses and^that it 
might be tirno t o up' ^ d'aTc Tc ^ r ltiq i TtTlcrrmsT ^lTe^s lEggi'es' t e'H^HiaTTFre" s 'cHo ol s . 
geT "away f rom"'sti^cc)Fv^ailo rm s and not submit to a ritual. I nvite the 
students to sav what they . w ant - if laudatory that would be fine. We 
shouldlTnder stand though what we are doing that is correct. The format s 
should b e^q h^JELO-d -aJLl^ bufc the f Q-^ma t. .ghould . 

never be mechanica l. Timing of critiques in the long course should be 
studied. The use of critiques are a good way to judge the performance 
of speaker s and if they arc not good we should be able to do something 
about it. Further, Mr. Cunningham stated that c ritiques can.be .com- 
mented upon in End-of- Course r eports or a s h as been done _h_y_- 


mar ginal comments made on the critiquc-i-Qxm..ft.s.£,L £. _ As another method 
oT7eedback7~ IZ ""^[ su ggested that DTK drop in to some of the classes 


to monitor presentations. 


also suggests that because of the 


complexity in some of the course construction in SUS that DTR call first. 
Mr. Cunningham stated that he has not had the time to drop into as many 
of the classes as he would like. He stated that although the value of 
critiques varies a great deal he has developed a respect for some staff 
instructors through critiques. 


The DDTR explained the position of the front office that the benefit 
of critiques is to tell where to improve and if we are coming across we 
ought to know it. The front office is quite flexible on format and invites 
changes. Mr. Cunningham remarked that when the briefing is given to 
the DDs that we need to have a goal in explaining the feedback we do 
get from student critiques and the End-of- Course reports as how improve- 
ments are made. The DDTR added that the critiques have a political 
value using as an example the briefing of the OMB . 


25X1 A 


25X1 A 
25X1 A 
25X1 A 


t As a part of the discussion of student critiques, it was again emphasized 
by Mr. Cunningham that it sh ou ld be the responsibility nf J:he ^nhie. Lln s_tn.LC.- 
| tor or course coordinator to follow up on student critiques, e. g. , in the 
' longer courses that are held J j that maybe after six months 

conversations should be held with students so that they can be a bit more 
objective than when critiques are written immediately following the course 
when emotion may dictate the type of critique that may be submitted. 


Jsuggested that instructors from 


take part 

in the debriefing of CT's who have returned from their initial overseas 
tour. wondered if these debriefings had had an effect. 


Mr. Cunningham remar k cd that in several instances the comments were 
made by these CT's th at t h e wives would h a ve bene fitted from some 
training. 
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25X1 A 


The critiques should be more productive and experimentation is 
certainly in o rclor along with special critiques such as interviews after 
students have left the cou r se. The critiques and End-of- Course reports 
"should be synchronized. It sh oul d be a balanced system. In interview- 
ing students aft er the course, c, g. , BOC- students after six months, 
not all students have to be interviewed, just be sure that all have an 
opportunity to be heard. 


reminded the Council members that critiques are only 

one method of feedback and in writing End-of- Course reports that as 
many methods of feedback as possible should be incorporated in the 
report. Each instructor should take a conscientious look at what is 

available to him in the way of critiques. asked Mr. 

Cunningham if he thought that the course chairmen were paying attention 
to critiques. Mr. Cunningham replied that he didjiot think so although 
he could not cite a specific case. 


$ 




also asked the DTR if he still wanted to get all the 

crit: 

iques, Mr, Cu 

nningham said he would want all the critiques that 


are written bearing, in. mind Chat critiques should be adapted to the needs 
of each course. 


2. The Catalog was the next agenda item. 


The DDTR explained that it is a unanimous opinion that the Catalog 
is out of date. In line with setting up the Catalog a discussion ensued as 
to prerequisites. Mr. Cunningham was curious as to how large a prob- 
lem it is in enforcing prerequisites. The Chairman stated that when 
confronted with a problem concerning prerequisites we invariably lose 
the argument. Mr. Cunningham resolved that we have to be sure of our 
ground and back it up. In letting up on the prerequisites we have to 
examine what it does to the course not the individual;for example, ad- 
mission to the AOC should be limited to operational people who are 
scheduled for an overseas assignment. When a discussion arises it 
should be between the schools and not ISS. 


25X1A 
25X1 A 


explained the discussion he has been having with 

wanted to know who made 


about the AIS. 


the decisions on enrollment. Mr. Cunningham said that we are respon- 
sible for the quality of the course and that rigid quotas from the 


r > * ’1 vt 


Curriculum Council 

... 12 November 1971 
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stated that the turnover of TO's within the Directorates causes 
problems in registration and also the lack of knowledge of OTR procedures. 
It was suggested that the Curriculum Council should meet with all TO's 
at the same time and contact should bo made on a selective basis with TO's 
regarding the Catalog. 


The subject of cut-off time in registering for courses was not discussed. 
The minimum cut-off time is a week for all courses, but constantly violated. 

said that he needs to have a minimum of two weeks in order 
to set up seminar rooms, assignment of counsellor briefings etc. The 
Chairm an said that we s hould get tough about it once we have set cut-off 
dates. 


let him work it out with the branches. 


CS/TRO, should be given the cut-off time and 


Notifications of deadlines o n courses should be sent by letter to the 


Training Officer, 
in the Catalog. 


said that a statement should be included 


| | brought up the subject again of prerequisites for entrance 

into BOC -- I WA, Records I & II, how to be made clear in the Catalog. 

|says it is not clear on Records I & II - only a recommendation. 

The Chairman will take this up with the DTR. 

3. The Chairman was curious as to how the school chiefs were now 
handling student critiques. Various techniques are being used -- from the 
blank sheet to the question form. The format of the critiques are being 
structured in line with individual course needs. 




Curriculum Council 
10 December 1971 
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Curriculum Council ivl noting 
13 April 1972 
(1000 - 1230) 


In Attendance: 


1. 


opened the meeting by telling the Council that he 


had metnioned to Mr, Cunningham the idea of a precis and a notebook of 
courses, Mr. Cunningham thought that something along these lines should 
be done, Mr. Cunningham also wondered how the Curriculum Council 
could operate without some sort of basic guidelines. There still was con- 
siderable dissent as to what the exact purpose of such precis or notebook 
would serve. It was determined after considerable argumentation that 
the precis or notebook would serve more than just bureaucratic purposes 
and that it would provide a continuity for changes in personnel and also a 
ready reference in case questions are asked about a particular segment 
from senior officials. The Chairman told the membe rs that the matter 
o f format for the notebook would be left up to each Cou r se Chairma n in 
confere nce withjT~ | No deadline was placed on comp le tion of 

the notebook but it s h ould be finished in CY 72 , . ] will report 

on the development cf the notebook on a periodic basis. | 
will also check out the Course Chairmen on a periodic basis. 


25X1 A | 


2. The subject of lack of reporting when changes are made in courses 
was next discussed. It was determined that only substantive changes need 
to be reported. If the change only concerns calendar changes no reporting 
is necessary - this can be handled at the School Chief level. The front 
office needs to know when substantive changes are planned in order to 
keep its own records straight e. g. , additions or deletions from the 
Catalogu e. The planne d substantiv e changes shoul d be repor ted to Mr. 

whojin turn will report i t to DPTR . A decision will then be made 


as to whether the proposed chang es should b e a matter o f di sc uss ion for 
the Curriculum Council. 


.Along these lines, 


intends to bring"|-^j||si 


the OJG in front of the Council. 


h ' - ChirrietxLum Council 
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25X1A 
25X1 A 


STATSPEC 
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7> Records Management Courses - The prospective retirement 


will, eliminate the recently initiated training program 

in Records Management Files, Forms, and Disposal -- unless 
alternative resources are developed. The Council should decide 
what action is to be taken in this connection, i.e. , drop the program, 
determine substitute instructional resources, etc. 


8 * Changes in Course Titles - The renaming of two of the 
A gency s four Directorates technically makes the names of several 
OTR courses obsolete. None of the core courses is affected, but 
othei s which are include CS Records I. II. ITT; Surmnrt .Sprviroc 


Orientation ior JJDb&T. In addition, all course descriptions which 
contain the former names of Directorates are affected -- these total 
approximately 25 courses. Areas for action include the Catalog, 
Schedule of Courses (now being compiled), and other OTR issuances 


Advance S eminar on China — Review of a new program 
proposed by SIWA need, objectives, process of design and develop- 
ment. Briefing by C/SIWA. 


10. Studen t Critiques - How and what should we be soliciting 
from the students, and what do we do with what they give us? 


11 • Transfer of Ops Support Course 


mends that the O 

ss Support Course be transferred 



C/OS does not favor the proposa 



i except within 


I ■ ■ I I X — — - ‘ ~ Jt-' * VVJ.CJ.iA 

the possible context of putting all operational training under one 
jurisdiction . (Memoranda attached) 


25X1 A 


_ 12 • ISS Me dia Services - C/I SS, per agreement at the last 
meeting of the Council, has solicited from School and Staff Chiefs 
theii projections for use of film, TV, and visual aids services 
over th e next six months. C/ISS also has received requests from 
ISD and | | for major film productions. He requests guidance on 

establishing priorities . 

Committee on Classroom Design Technolo gy - A report of 
this committee's first meeting is attached for review by Council members. 


Curriculum Council Agenda 
12 April 1973 


I ' ■ ' " ' ’ V> 
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For Curriculum Committee Meeting, 25 October 1973 


The following tabulation, based on a quick analysis of the course re- 
ports for three recently-conducted OTR courses, shows the kinds of inf or- . 
mation conveyed and the location of each kind of information within the 
total course report "package". All "packages" were alike to the extent 
that they consisted of the body of the report and accompanying attachments. 


Kind of Information 

Course X 

Course Y 

Course Z 

1. Course Objectives 

— 

Att. B 

Para. 4 

2. Roster 

Att. A 

Att. A 

Att. I 

3. Schedule 

Att. D 

Att. B 

Att . J (plus B-K) 

4. Background Notes for Speakers 


— 

Att. I 

5. Student Body Statistics 

a. Dropouts, additions, part-time 

Para. 1 

Para. 2-3 

Para. 1 

b. Agency /component breakdown *• 

Att. B 

Para. 1 

Para. 1 

c. EEO Data 

Att. B 

— 

— 

d. Grade range/avg. 

Att. B 

Para. 1 

— 

e. Age range/avg. 

Att. B 

— 

— 

f. Length of service/avg. 

Att. B 

— 

— 

6. Changes 

a. From- previous course 

Para. 2-6 

Para. 4 

— 

b. Planned for next course 

Para . 3,4, 

Para. 7 

— 

7 . Admin Probs 

7, 8 

Para. 5,7, 
8 

Para. 3,6 

— 

Para. 3 

8. Instr. comments re guest speaker 

Para. 5 

— 

9. Instr. comments re class 

Para. 2 

Para. 6 

— 

10. Student Reactions 

a. Instr. summary of — 

Para, 2-6 


Para. 3 

b. Achievement of Objs: stats 

— 

— 

Att . A 

c. Verbatim Critiques 

Att, C 

— 

— 
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24 October 1973 


MEMORANDUM FOR: OTR Curriculum Committee 

SUBJECT : Use of Audio and Video Cassettes/Seif-Study Program 


1. The use of audio and video cassettes as an educational system in 
OTR presents at least the following four options: 

e,, 'Formal... structured courses or Programs consisting cf r, pro— 
series ex: courses developed. 6 ud.lo and/ <:r v: 

eabsetr.es xu Eeduiar, sequential form as the tunaaiiientai teacni.o.g 

medium. Such a course or program using this new technology should 
be developed and conducted by the appropriate OTR component. The 
cassettes may be purchased or produced by OTR. The course would be 
a regular OTR offering utilizing new educational techniques and 
equipment. It would not be a "self— study" course or program. 

b. A variation of the above would be to conduct videocassette 

courses or programs for a specific office or group at a time and loca- 
tion of their choice and provide an OTR substantive specialist as the 
course coordinator, discussion and seminar leader. This variation 
also is not truly a self-study effort, but rather a special training 
service using new systems and offered by OTR. , 

c. A Self-Study Program operated by OTR would enable individual 
Agency employees on their own initiative and convenience before, 
during, and after duty hours to take courses and attend lectures by 
the use of video and audio cassette systems. For each cassette 
course offered, supplemental material such as reference book, work- 
book, and leader’s guide would be loaned to the student. Utilizing 
substantive specialists in OTR and the Agency, tests and final com- 
prehensive exercises could be developed for some of the courses to 
measure and verify achievement. Some of the workbooks accompanying 
cassette courses also show the amount of improvement or accomplishment 
Credit would be given for the training if requested by the student and 
a satisfactory level of achievement is verified. The testing or veri- 
fication of achievement would be done by OTR. In developing the Self- 
Study Program, determination will be made in conjunction with substan- 
tive specialists in OTR and the Agency of which cassettes shall be 
included, and also what cassettes should be produced by OTR. The 
Program Officer is not expected to be a substantive expert in any, 

let alone all, of the fields which may be covered eventually by 
cassette programs. 
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d. An additional Self-Study Program configuration providing 
increased capabilities could be obtained by having mobile self- 
study units. With this feature, the equipment, cassettes, work- 
books and guides would be loaned to an office for group presenta- 
tions and discussion; or made available to an office for a period 
of time-one week to two months — for individual use by members 
of that office at their facility. Whether for group or individual 
use, the course would be conducted entirely by the requesting 
office. 

2, OTR needs to decide which of these options, if any, or combination 
of options it desires as an objective. At the same time, it must make the 
necessary commitment of money and manpower to achieve a degree o*. success. 

If options a. or b. are selected, either the equipment now planned for use 
in the Self-Study Center would instead be used for developing and conducting 
regular OTR course offerings, or additional equipment will have to be pur- 
chased. Options a. and b. also require the assignment of OTR instructional 
personnel to develop and conduct thji courses. Options c. and d., a Self- 
Study Program, would, in addition to a Program Officer, utilize on a limited 
part-time basis OTR substantive specialists to develop tests and possibly 

to administer the tests. OTR personnel may also be used to produce video- 
cassette programs for self-study (educational TV). If a Self-Study Center 
is put in operation at Headquarters and receives enough usage to merit its 
continuation, one full-time training assistant located in the Center will 
be required to operate the Center. 

3. The DTR has expressed a desire to develop courses that would be 
in option a. His comments made on the routing sheet of my 19 September 
memorandum to C/PDS on the subject of the OTR Self-Study Program were, 

"We need to develop the concept of a course , or courses using this tech- 
nique, i.e. a series of films that essentually cover a unity of management 
thought or problems. For examples The Principal Ingredient s of Efficient. 
Management 

a. MBO 


h , De c j 5 :i .on king 

c- Pr o i f xari a ggaicT* t 

What I'm trying to get at is a package that holds together— is a unity— 
begins, middle and ends — not just a bunch of disconnected tapes!" In a 
note to C/PDS on the Self-Study Program the DTR stated, "I think all of 
us will agree to sit in a room and see the self-study programs or packages 
that we may have would be useful, but certainly not as useful as would be 
the case if the films were to be analyzed and discussed by either a small 
group of two or three, or even a large group, maybe up to 12 in a seminar." 
It appears that what the DTR wants is not exactly what we have envisioned 
as a* "Self-Study" Program. If the course or "unity" of courses are to be 
developed using both videocassettes as a teaching medium and personnel 
to conduct seminars for discussion and analysis, the development task 


Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 0007-0 


-RDP78-O6ftPAOO«*0OO1 0007-0 


should be assigned to the OTR component responsible for the specific subject 
matter (the Functional Training Division's Management and Administration 
Training Programs in the case of Management Course cassettes, for example). 
The development and running of the new courses that the DTR desires and 
a true self-study program for the Agency will probably require additional 
OTR funds for more equipment. We need to define what the OTR objective 
is and allocate the necessary training resources. Then and only then can 
we effectively proceed to accomplish something new. 


Approved ter Reltese 2002/11/01 


CIA 


STATINJL 

Self-Study Program Officer 
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26 October 1973 


MEMORANDUM FOR: Chief, Curriculum Committee 

SUBJECT : Coverage in the Curriculum of International 

Economics, Ecology and Demography 


Listed below are the presentations on International Economics, 
Ecology and Demography in recent runnings of our various training 
courses . 


(A) SENIOR SEMINAR No. 


"World Population Problems 
STATINTL (Demography) 


"The World Economy: A New 
Dimension in U,S. Foreign 
Policy" 

(International Economics) 


"U.S. Business Atroad" 
(International Economics) 


STATINTL "Environmental Crisis" 

(Ecology) 


, from 21 January - 23 March 1973: 


22 Feb. 1973 
2 hours 


Deane Hinton '23 Feb. 1973 

(Council of Int'l 2 hours 

Economic Policy) 


27 Feb. 1973 
2 hours 


12 Mar. 1973 
3 hours 


(B) ADVANCED INTELLIGENCE SEMINAR (AIS) No. 12, from 
30 May -~19 June 1^731 


"Economic Power and International Irwin Tobin 
Politics" (Int'l Economics) (Treasury) 


4 June 1973 
1 1/4 hours 
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STATINTL 


"National Priorities and $" 
(International Economics) 


"Multi-Nationals: How 

Important and Who Benefits?" 
(International Economics) 


6 June 1973 
1 3/4 hours 


7 June 1973 
2 hours 


(C) INTELLIGENCE IN WORLD AFFAIRS #2-74, 10-28 September 1973: 


Panel: Europe 

(International Economics)* 


21 Sept. 1973 
2 hours 


21 Sept. 1973 
1 hour 

21 Sept. 1973 
1 hour 


"Issues in the Mid-East"* 


ONE 26 Sept. 1973 
2 hours 


(D) MIDCAREER COURSE #37, 22 August to 5 October 1973 


"New Emphasis on Economic 
Research" 

(International Economics) 

"Prospects for the Middle East" 
(International Economics) 


"The U.S. in the World Economy" 
(International Economics) 


Maurice Ernst 13 Sept. 1973 

OER 2 hours 


26 Sept. 1973 
Spec. Ass't to tiie 1 1/2 hours 
DC I 


Charles Cooper 3 Oct. 1973 
Deputy Ass't to 2 hours 

the President for 
International 
Economic Affairs, 

NSC 


* Approximately half of this presentation dealt with international 
economic matters. 
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"International Politics and 
Environmental Problems" 
(Ecology) 


Christian Herter 28 Sept. 1973 
Spec. Ass’t to the 1 1/2 hours 
Sec. of State 


"Energy: Challenge of the 

70 ' s" 

(Ecology) 


James West 4 Oct. 1973 

Staff Ass’t to 2 hours 

the Ass * t Sec . 
for Mineral 
Resource, Dept, of 
the Interior 


(E) LATIN AMERICAN SEMINAR 


"International Relations: 

Emphasis on South America" 
(International Economics) 


"The U.S. Role in Economic 
Development: The Latin 

American Reaction" 
(International Economics) 


24 Sept. 1973 
3 hours 


tics 


13 Nov. 1973 
3 hours 


"The Church and Demographic 
Problems" 

(Demography) 


<m T 


30 Oct. 1973 
3 hours 


(F) INTELLIGENCE PRODUCTION COURSE 


Session on International 
Economics 

(International Economics) 


Visit to Office of Economic 
Reporting 

(International Economics) 


Robert Morris 26 April 1973 

Chief of Staff, 2 hours 

Council on Inter- 
national Economic 
Policy 

| 2 April 1973 

OER 4 hours 

STATINTL 
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(G) CHINA FAMILIARIZATION COURSE 3-73 


STATINTL 

"Geography and Natural 
Resources" 

(Ecology and Demography) 

STATINTL 

"Economic Development and 
Exploitation of Resources" 
(Ecology and Demography) 


27 Mar. 1973 

USUI 1 hour 


tmr 


28 Mar. 1973 
1 hour 


STATINTL 

"The Social Revolution" 
(Ecology and Demography) 

STATINTL 


UTJT 


28 Mar. 1973 
1 hour 


Chief, intelligence Institute 
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1 November 1973 


MEMORANDUM FOR: Members of the Curriculum Committee 
SUBJECT: Equal Employment Opportunity Statistics 


1. In the last Curriculum Committee meeting, there was a 
question about the need for including EEO data in end-of-course 
reports. The Chairman requested information on what EEO data is 
currently being compiled by the Registrar's office. 

2. The Registrar routinely submits the completed class 
rosters for computer input. In OJCS, the rosters are matched 
with a key, and the result is an EEO computer run which is 
issued quarterly. The run shows training, internal and external, 
taken by white females, black males and females, other male 
minorities, and other female minorities. Copies are sent to 

the EEO representative; a copy is retained by the Registrar. 


STATINTL 




ADMINISTRATIVE - r\: - •,£ y, 

Approved For Release 2002/11/01 : CIA-RDP' 

11111 


A0001 0001 0007-0 



STATINTL 


Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 0007-0 


24 October 1973 


MEMORANDUM FOR: Director of Training 

SUBJECT : Evaluation of the "Information Science for 

Managers Course," 15-19 October 1973 


1. The number of applications for this course were limited 
to about seventy-five, through the use of quotas. Thirty-five 
students were enrolled, of which 5ix dropped at the last minute 
because of the Mid-East Crisis. The twenty-nine remaining con- 
sisted of twelve CIA; nine NSA; five DIA; and one each Army, Navy, 
and Air Force. * 


2. The effectiveness of information science instruction 
begins to decline when students number more than about twenty, 
and with more than thirty the decline is significant. Produc- 
tivity and the demands for this kind of training are very strong 
arguments for accommodating as many students as possible. The 
optimum trade-off seems to be about thirty students. Classroom 
912 is adequate for as many as thirty-eight students. However, 
the GE terminal room is very crowded with thirty students and three 
instructors. This crowding could be eliminated by removing the 
light parti tion which separates the t erminal room from the office 


now used by 


It would also be necessary 


to move one telephone and one electric outlet. This change is not 
urgent, but it would contribute to more effective instruction. 


3. The subject of student course evaluations warrants some 
thinking and possible revision, perhaps for the whole of OTR as 
well as the ISTP. The main thing they tell us is the popularity, 
interest, and entertainment value of individual presentations and 
the course as a whole. They do a poor job of indicating the utility, 
value, applicability, and pay-off of our courses to the Agency and 
the Government. Since the students evaluate instruction most heavily 
on the basis of interest and popularity there is a temptation for 
instructors to design their courses to meet this criteria. The 
extensive use of dramatic guest lectures, senior executives, and 
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tours is the easiest way to make a course most interesting. 

Unfortunately it is not the way to teach skills and methods 
which can be applied to the management of intelligence or 
the work of intelligence. The dominant or exclusive use of 
student evaluations places an inappropriate premium on popu- 
lar and interesting presentations. It discourages the presen- 
tation of subjects which are inherently difficult, dry, or 
tedious, even though they may be very useful. The ISTP has 
resisted this temptation and makes only a minor use of guest 
speakers. However, there remains a need to develop some 
means of evaluating the utility and applicability of course 
content, and student evaluations simply cannot do this. 

4. Over the past three years we have used many different 
evaluation forms. I favor the simpler- forms , in the belief that 
all they really tell us is how well the students liked the course. 

The more detailed forms simply tell us how well they liked 
individual instructors or subjects. Despite the .limitations of 
student evaluation, their evaluations of this course are grati- 
fying. Question three on the attached form attempts to measure 
the course utility and value from the students' view, I suspect 
that all it really indicates is popularity or interest. 

5. Additional comments were requested on a second page. 

Nine of the twenty-seven evaluations included spontaneous 
written compliments to the staff, such as: "I give four stars 
to the staff," or, “The individual performance on the part of 
most instructors was outstanding." There were only a few mild 
complaints on minor points, such as the student to whom "scientific 
data banks were not applicable," or the student who "hadn't used 
math for twenty-five years." Most rewarding was the comment, 

"The course presented a number of techniques which I will take back 
to the office and use almost immediately." That is our major 
objective, and we are successful to the extent that we accomplish 
it. Perhaps it should be a major objec tive imall training, 

STATINTL 

Course Director 

Atts as stated 
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19 October 1973 

INFORMATION SCIENCE FOR MANAGERS 

COURSE EVALUATION 7 


Briefly stated, the objectives of this course are: 

° To acquaint managers and intelligence 
professionals with the terminology and 
basic techniques of Management Science. 

° To identify the capabilities, limitations, 
and applicability of Management Science 
methods to the functions of management and 
the processes of intelligence. 

•+ * 

0 To improve the ability of -managers and 
intelligence professionals to communicate 
with systems professionals, in the accom- 
plishment of management and intelligence 
tasks. 

1. Considering the limitations of time, how would you 

evaluate the course in accomplishing these int roductory 


objectives? 

Outstanding 

Excellent 

Good 

Fair 

Poor 

(/-/ ) 

at) 

ID 

(/) 

( ) 


2. Would you recommend this course to any of your colleagues? 

Yes No 

m) ( / ) 

3. In comparison with other training, how would you rate 

the utility and value of this course? *'■«' 

Very Useful Moderately Useful Nice to Know 

{/?) (/s.) . . ( / ) 
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Bruce Monroe 

Vice-President of / XS GROUP (In- 
Structional Systems C roup) ; teacher, con- 
sultant, and administrator. Long Beach 
State. 

James B. Quinn 

Educational psyci otogist. Mental 
Health Development Center, L.A. 


From kindergarten through higher education the r.ew “thing” in 
education is writing bolter objectives, hveryone — teichors, counselor*, 
administrate rs — will be expected to do so. The begim ng teacher ca :i 
anticipate b-dng on the job only briefly bet ore he is isked by someone 
important, ‘ What are your objectives for this unit? Oi course? Or class? 

Or student?” With the rapidly increasing requirements for evaluation 
and accoun ability in education, the teacher also can ?xpect that Ins 
ability to set objectives will play an increasing role in hi attempt to do 
the job expected of him. 

Why? . ; 

How come the clamor for objectives, objectives, objec lives. 

What is so intrinsically good about educational obj-'Ct ves that causes 
... legislators, boards of education, superintendents, program developers, 
principals, and other instructional specialists to dema id them? tins 
chapter attempts to explain, “Why Objectives? • 

Moreover, if objectives are so important to educators, they must 
have characteristics which distinguish them ; in des liable ways. A 
significant question for most teachers becomes. What characterizes a 
£>d M objective? llow should objectives be written? 

3 This chapter attempts to explain, “How Objectives?” 
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| . , I 

The R<i!e of Ob>ec(ives in Curriculum and instruction 

j . . ...... I 

An emerging axiom in education holds Mint by good objectives 
educators mean objectives which can bo measured. “Good” objectives, 
j then, equal measurable objectives. Measurable objectives arc in fact 
] measurable when they indicate to the instructor how he can determine 
whether the learner has accomplished the objective. Or, the instructor 
should know how Ire will assess the achievement of a given objective 
| simply bv rending the objective. 

It follows that the most fundamental role of the educational 
objective is to denote the measurement of student achievement, 
j Well-written objectives describe student achievement, a priority j 

1 which most educators believe has not been emphasized enough and 
increasingly should be emphasized as we plan instructional experiences. 

Because objectives arc so strongly bound to student achievement, 
they serve in several ways which educators find very useful. Educational 
objectives, for example: 

i j 

j 1 . Indicate the learning needs of students. 

j Usually, objectives result from an effort which has been 

termed “needs assessment/' Before writing objectives the " 

teacher asks, “What arc the learning needs of my students?” If 
Die answer is not apparent, or if evidence concerning learning 
needs is unavailable, the instructor is forced to perform “needs 
j assessment” tasks prerequisite to writing objectives. 

In another sense, objectives can indicate what a student 
needs to learn about a given educational or instructional unit or 
program. Such needs can be considered unrelated to, or 
independent of, a student’s previous learning or experience. 

They refer instead to the learning needs a student must acquire 
to demonstrate “mastery” of a given program or subject matter. 

I 

2. Serve as targets of instruction. 

By “targets” is meant the kinds and levels of achievement 
which help the instructor plan his instructional program, 
j Objectives loll the instructor where he wants to go; hopefully, 

: they will also enable him to plan ways of getting there. 

By indicating targets of student achievement, the objective 
or;cnt$ the instructor to an analysis which might indicate what 
kinds of prior student achievement are prerequisite to 
accomplishing. the objective. 

Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 000^0 




A pprov e d ' teas e 2002/11/01 . C I A-RDP70-002lW)0CHf000 1 0007-0 

By indicating a level of student achievement, the objective 
orients the instructor to considerations such as how long he will 
want the student to practice desirable tasks, and how to arrange 
inst rue tional c mpl i ascs . 

3. Guide the evaluation process. 

Evaluation is partly the process of assessing an output 

against a predetermined standard. Educational objectives form 
the standards of educational evaluation. Against the standard of 
the objective, educators arc able to make evaluative statements 
about the relationships among objectives, procedures, methods, 
and. conditions. Lacking objectives, the evaluation process 
reduces itself to an exercise in describing what happened during 
instruction and what was learned, but neglects telling us how 
y/hnt was learned relates to what was expected. 

4. Reflect student gains. 

Objectives serve as useful rcinforcers of student 
achievement. The student who knows what is expected of him 
also knows when he has met those expectations. Objectives help 
instructors HU a void which has hounded education for years, 
the void of feedback. 

Objectives which monitor the sequence of an instructional 
program also monitor the student's readiness to proceed to the 
higher order concepts and principle* of the program. Students 
often fail because they lack the learning prerequisite to a greater 
achievement. Objectives which arc prerequisite to other 
objectives can be used to pace student learning. 

5. Reduce student competition. 

Objectives are impersonal. Their place of distinction in an 

instructional program permits students who are accustomed to 
competing against one another for recognition and rewards to 
work cooperatively and compete against an impersonal 
standard. In an ohjectivcs-bascd sequence of instruction the 
learner succeeds, not by placing high in a student rank-ordering, 
but by conquering the instructional objectives. 

Objectives have value for educators because they serve many 
functions which educators believe contribute to that elusive 
rainbow, “quality education.” Objectives describe student 
achievement, indicate learning needs, serve as targets for 
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i I! 'f, ir bf ‘‘ ad f ,t W!,sc ‘ however, objectives arc important 

OvCJuMv 1 1 ? c \ j re tpc caialysi* oi educational change. 13y valuing 
objectives .50 highly, educators arc forcing teachers, counselors, 
and auministrators to function in more precise ways, leaving less 
: learning to chance. ' " 

The single greatest change of the past decade has been the 
move in education from a process-oriented profession to an 
outcome-oriented profession. Measurable objectives reflect 
concern for the student and Ins accomplishments. It says, in 
c(Tcct - that l ,r0 jjrants, methods, instructois, materials, and their 

cumulative worth are “good" only to the extent that they help 
the student. 

fne ultimate value of objectives, the ultimate “why", is that 
objectives assure that the point of departure for planning and 
doing in education will be education’s clients, its students. 


Writing Meaningful Objectives 


The cotton of the meaningful objective is, of course, an abstraction. 
Objectives are meaningful wh.cn they consist of characteristics which 
have meaning to someone, in this case educators such as teachers 
instructional specialists, and evaluators. 

As has been mentioned previously, meaningful characteristics of an 
educational objective arc appropriateness and measurability. 

Measurability docs not mean able to be assessed by a written test 
The concept is more genera! and refers to an ability to determine 
wnethcr the objective has been accomplished. The methods for 
inea, urmg objectives are many and include observation techniques, 
sc“ -reports, role-playing, rating, scaling, interviewing, written tests, or 
any tec, unique which promises to render a valid assessment of an 
objective s accomplishment, or lack of accomplishment. 

Imminent labels for these measurable objectives include 
performance objectives, behavioral objectives, instructional objectives, 
r . d • 'j o w-'j ~ w t i v e s , and curricular objectives. due common 
Oe.uu.n. Uior jmumg mesc ;s iiieir measurability, and so the term 
measuiame objective lias emerged as most inclusive. 
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I agree that vsritmg meacuubie objecmcs. in difficult. 
Woil-writtcn educational objectives require, first, a way of thinking, 
and, second, practice. The remainder of tins chapter will be devoted to 
these requirements. 

Weil-written educational objectives specify three kinds of 
information: 

1. Information about the intended outcomes of instruction stated 
in observable terms. 

2. Information about the level of achievement of the outcome. 

* 

3. Information about the conditions of evaluation under which 
one might determine whether the level of achievement has been 
reached. 

Consider briefly the following objective of a course in “creative 
writing/’ ' 

“The student will appreciate the imagery of Hemingway’s writing.” 

Apply the specifications- of measurable objectives and ask whether 
this objective is well-written and meaningful. 

Why? Why not? 

If you determined the objective was well-written, you concluded: 

1 . The objective, as stated, is measurable. 

2. The outcome of instruction is specified in observable terms by 
“student appreciation.” 

3. A level of “student appreciation” is specified. 

4. The conditions under which the achievement of “appreciation” 
can be assessed are specified. 

The objective, as stated, meets none of the specifications. Although 
the outcome of instruction .is “student appreciation,” the objective 
does not indicate what the student must do or say to demonstrate that 
appreciation. As will be pointed out shortly, the use of abstract nouns 
sue!) as “appreciate” do not allow verification, and thus are 
inappropriate. 
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After rcaP(E Hemingway’s short story, “A Clean WefTTighted 
Pbee, ” the student will list all the adjectives which appeal to more than 
one of the semes. 

Consider the characteristics of the restated objective. 

First, the objective is measurable, because we can deduce from the 
student’s listing if he has accomplished the objective. 

Second, the outcome is specified in observable terms, “the student 
will Us: adjectives which appeal to more than one sense.” 

Third, the level of achievement is specified, “the student v/ill list all 
the adjectives.” Fourth, the conditions of evaluation are specified, 
“after reading Hemingway’s short story, ‘A Clean Well-lighted Place’.” 

Determining Student Outcomes 

The first characteristic of meaningful objectives is they describe in 
observable terms the intended outcomes of instruction. Outcomes refer 
to what the student will have achieved or advocate after instruction. 
Outcomes, then, arc the ends of instruction as opposed to the means of 
instruction. 

The process of writing objectives usually begins for the instructor by 
his determining what kinds of student performance the instruction 
should elicit. There arc three: knowing, feeling, and doing. As a result 
of instruction, that is, the student should know something he did not 
previously know, or he should feel differently from the way he did, or 
he should demonstrate new behavior, or any combination of the three. 

A noted learning psychologist, Benjamin Bloom, has called these 
three kinds of educational outcomes, “domains” - the cognitive 
domain for knowing, the affective domain for feeling, and the 
psycho-motor domain for doing. One consideration for instructors 
then, is the kinds of outcomes they intend the instructional program to 
effect. 

While the literature of educational objectives has emphasized the 
technology of writing objectives, a similar emphasis has not been 
afforded the different kinds of objectives instructors can specify. 
Withm each of the three objective domains, for example, various kinds 
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instructors arc advised strongly to familiarize themselves with Bloom’s 
taxonomies of educational objectives, and set various kinds of 
objectives in the various domains. 
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Setting Levels of Achievement 

By specifying a level of achievement for his objectives the instructor 
indicates how well he expects the objective to be achieved. 

In'thc creative writing objective, the learner was expected to identify 
all the words appealing to more than one sense. By requiring that all 
such words be identified, the instructor set a standard against which the 
learner’s performance could be evaluated. Objectives which lack such 
standards arc difficult, if not impossible, to evaluate. 

The levels of achievement specified for particular objectives result 
largely from instructors’ value judgements about the importance of 
objectives. High levels of achievement are recommended to the extent 
the objective is seen as prerequisite to the learning of other higher-order 
objectives. Lower levels of achievement are recommended to the extent 
the objective: (1) is not a prerequisite objective; and (2) requires a 
higher-order skill, such as the cognitive skills of evaluation and 
problem-solving. 

Level of achievement statements may be quantitative or qualitative, 
or both. They may specify a rate or percentage of learning, such as SO 
per cent or 60 per minute, or they may specify certain qualities or 
characteristics of the performance, such as “draw a straight line” or 
“argue politely. ” 

Qualitative levels of achievement, it should be einpahsized, should be 
characterized by descriptive words such as “all,” “ten,” “straight,” or 
“red ” as opposed to abstract words like “good,” “highly,” strongly ” 
“very much,” or “very well.” Descriptive statements arc ones whose 
validity can be observed by one or more of the senses, and can be 
wuuabtuu tu ausiruct suucunenis whose validity must be inferred by 
the observer. 

The following three sets of objectives each contain a descriptive 
statement ami an inferential statement. Decide which statement from 
each set is descriptive. 

1 The student will grasp (!:<; ■iii'u.iiice of the Civil War. 

^ stucent will write a 500-word essay in which tie will discuss 
why '.he (hvii V.'a r was significant. 


hPP 
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]. The sluderj! will explain (he principles of chemistry. 

2. The student wilt become familiar with the principles of 

chemistry, 1 


] , The student will appreciate the beauty of an aesthetic 
experience. 

2. The student will report the beauty of an aesthetic experience. 

The reader who chose the statements containing the verbs “write,” 
“explain,” ai -d “report/’ probably understands that a descriptive 
statement is characterized by behavior which can be observed. 

Prescribing Conditions of Evaluation 

Conditions of evaluation refer to the circumstances or situational 
factors under which the student is to demonstrate his achievement of 
the objective. They may refer to a given time period, or the 
instructional materials to be used, or some kind of stipulation or 
contingency. Conditions of evaluation usually limit or designate the 
boundaries of the performance. The creative writing objective limited 
the student’s references to a single short story. 

The conditions of evaluation specified in the objective should be 
conditions related to the learner’s ability to achieve the objective. 

Again, tire importance of the given objective must be considered 
before the conditions of evaluation arc specified, so that more 
important objectives should be characterized by stricter conditions of 
evaluation. 

Conditions of evaluation should not be confused with conditions of 
instruction. Conditions of evaluation relate to student performance 
after instruction; conditions of instruction relate to teacher and student 
■performance before or during instruction. Measurable objectives should 
not include statements .which describe instructional procedures, or 
teaching methods, or the ways in which the student will study. These 
are conditions of instruction. 

Practice Makes Perfect 

Now consider the following objective of nursing instruction: 
u , , . calculate the drops per .minute, accurate within five drops per 
minute, using an accepted formu'a and chart. . 
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What b ihc expected level <d achievement ? 

What conditions uf evaluation are specified to he held constant as we 
look for titat level of accomplishment? 

You should have noted that: 

The intended outcome is: . .calculate the diops per minute. 

The expected level of achievement is: “...within five drops per 

minute. . . 

The conditions of evaluation arc: “...using an accepted tormina 

and chart.” 

The reader should use the form of this objective, (using outcome, 
level of achievement, and conditions of evaluation in that chronological 
order,) and write five measurable objectives relevant to his own subject. 

' The reader should next write three measurable objectives for 
instruction in “creative writing,” a subject for which the writing ot 
measurable objectives would be considered by many instructors as an 
elusive, anu perhaps impossible, task. 

A Validity Check 

One purpose of writing measurable objectives is to communicate to 
learners the desired results of instruction. Measurable objectives are 
valid only if learners who read them know what is expecteu of diem, 
how well they arc expected to perform, and under what conditions 
they will be expected to perform. 

An instructor can validate his objectives by asking others — 
colleagues, students, friends — to read his objectives and to write down 
the intended outcomes, achievement levels, and conditions oi 
evaluation communicated to him by the objective. The answers should 
match the instructor's answers to those questions. 

This simple procedure could be tried economically with a sample of 
three to five people. If disagreements arise, objectives need to be 
^written auu mod out again, until those reacting to objectives perceive 
them as communicating a clear and common message. 

The “Objectives 1 ’ Debate 

Many educators object strongly, both for philosophical and 
methodological reasons, to extensive use of measurable objectives for 
purposes of curriculum design and instructional planning. They 
ch alienee measurable objectives for being denunum/.mg, impersonal, 
skdboi'icnted, narrow targets, time -wasting, and trite, among other 
tkmgw The debate has been ragmc for sortV time." 
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The purpose here is not to review these objections. The challenge, 
however, has been met well by educators who see mensurable objectives 
positively. For an illuminating account of this debate the reader is 
urged to consult Popham (10). 

Fortunately, the swing to measurable objectives is gathering 
momentum. Hopefully, the role of measurable objectives in education 
will be expanded and new uses will be developed for them. 

Conscientious adult educators arc urged to think about other 
appropriate uses of measurable objectives, and to use them for planning 
instruction. 

Warm-Up and Orienting Activities 

Writing behavioral objectives is a way of focusing attention on- 
learner behavior rather than on mental process or on teacher behavior, 
if you focus on learner behavior , it will make your job as a teacher 
much easier. You will always know what behavior to reward, what 
behavior to change, and what behavior to evaluate. 

This activity is to get you more used to focusing on learner behavior 
rather than on mental processes or teacher behavior. 

EXAMPLE OF WHAT TO DO. Translate this educational goal into 
observable behavior: 

The learner will appreciate good music. 

Cheek each of the following learner behaviors that you believe show 
that the learner “appreciates good music.” 

goes to a classical concert. ■ ■ 

buys a magazine that tells about a classical music 

personality or event. 

tells about a book he has read voluntarily on some aspect of 

classical music. 

on a questionnaire, says he enjoys classical music as well as 

other types of music. (Questionnaire must not require 
learner to sign his name; otherwise, he will give the answer 
that will please you, rather than an honest response). 

tells the class about upcoming classical musical events. 

. (ells the class about classical musical events he lias attended, 

U\!s that he enjoyed the activity. 
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(NOTJ:: The learner must do the above voluntarily to mhjw 
■’ appreciation"; the teacher must encourage and reward this behavior to 
make It increase. IT the teacher requires the behavior, the learner will 
not be showing “appreciation" but rather that he will follow 
instruction.) 

If you checked all of the above activities, you have done very well. 

Each of these activities would be learner behavior that probably 
demonstrates “appreciation of good music." You can reward this 
behavior by telling the learner he has done well or by otherwise 
showing your appreciation for his activity. You cannot reward 
appreciation if it is only something the learner feels. You never know 
for sure what lie feels. A behavioral objective tells some behavior that 
yon can sec or hear or touch or smell or taste. (These are the only ways 
we human beings can observe our environment.) 

Which of the following verbs tell behavior that you can see, hear, 
touch, smell, or taste? Cheek the behavioral verbs. 


pass out pieces of 10. 
cake you baked 


jay “that’s very 12. 
good" 


js cooperative 

_ sits quietly 

_asks for help 

jsks for more instructions 

_ sandpapers a piece of wood 
until it is smooth 

_on a questionnaire, states 
that he likes math 


If you checked 3, 4, 5, 6, 8, 9, 10, 11, and 12, you have done well; 
these arc learner behaviors you can observe with your senses. 

I, 2, and 7 are mental or internal behavior; you cannot observe 
anutner person cioing tnesc unrigs with one of your senses. 

Some people believe in “intuition ” that a teacher can intuitively 
feel what is going on in another person’s mind. This may be true, but as 
yet we have no evidence that it is true. Teachers are better off not 
relying on intuition alone. Teachers are better off obsen’iftg responses 
of learners and using these observations to determine whether or not 
the student is learning. 

Here are examples of learner responses: 

1 . Wiitmg a paragraph or a sentence or a word. 
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2 R.m ITtv’ a word, sentence or story aloud, pronouncing the 
words correctly. 


3. Taking a test. 

4. Completing a questionnaire. 


Write two more examples of learner responses: 


Participants should read these aloud and be reinforced by the leader. 
.If tire person is incorrect, give him some more examples ora.ly an- as., 
him to select one. Reinforce his selection. 


Practice 

Here arc some examples of behaviors that show a person is learning 
to read. Add at least two more examples. 


Reading 

o seeing a printed syllabic and pronouncing it correctly 
o seeing 3 printed word and pronouncing it correctly 


o reading 


ting a sentence aloud, pronouncing the words correctly 


o drawing a line from a picture of a cow to the word “cow” to 
show he recognizes the meaning of the word 

o tcllim.’ the class about a story he just read 
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The lendcrklmuld have pai i ici pants io;r1 aloud their examples, 
reinforcing the correct ones. II' tire example is incorrect. the leader 
shiHild I’ivc two alternative responses and ask the participant to choose 
the correct one. Tiic leader should always encourage, never discourage 

with a negative response to a participant. 

If the participants response is a mental processor a teacher behavior, 
the leader should sav, “Thai’s a mental process; it cannot be observed. 

What could you sec’or hear?” Or, “That's a teacher behavior. What do 
you want the learner to do?” 

Below arc some more skills and abilities that we might want students 
to learn: | 

o analyze a poem 

o write an essay 

o compare and contrast two characters from stories. 

For each of these activities, however, tiic teacher must tel! what 
should be included in the essay, analysis, comparison. The teacher docs 
this with a checklist. This is part of the behavioral objective. 


Example 


The analysis of the poem should tell the following: D:d the author 
use similes, metaphorcs, personification, or onomotopoeia? Give an 
example of each of these literary devices that you found in the poem. 
Tell what the poet wanted to communicate to you by using each of the 
devices. 

If the learner has done all of these things, lie lias successfully 
achieved the objective. His analysis may not be complete, but ne is 
learning. The next assignment may add new items to the checklist. Tins 
is the method of “successive approximations.’' /his method says that 
everything the learner docs increases ins sku! or ability a little bit. Ac 
do not expect him to learn an entire complex skill, Mien as analysing a 
p«>em, all at once. Ife learns a little at a time. As he adds each oit to 
whai He has learned before, he increases his ability, leachers too often 
exrsat a learner to learn ewnyshme immediately. Skill in teaching 
m ernes breaking a complex skid into a series of behaviors, to be 
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List below at least three of the behaviors you think must be learned, 
one at a time, leading to this end result: 

The learner will be 3 ble to write a paragraph. 

Examples of “bits” of behavior that lead to achieving this objective: 

1. Select and read the topic sentence of given paragraphs. 

2. Identify a paragraph that needs a topic sentence. 

3. Writing a topic sentence for a paragraph that does not have one. 

Add two more skills that must be learned in the process of being 
able to write a paragraph: 


Leader should have them read aloud and reinforce correct responses. 
A Working Unit on Objectives: Questions and Exercises 

1. Which characteristics of measurable objectives do the following 

objectives lack? 

A. The learner will list instruments for measuring physical 
properties and human characteristics, 

B. Having been shown 10 drawings completed by Ids 
classmates, the student will discriminate between those 
which represent op art and these which represent pop art. 

C. When presented with three hypothetical descriptions of 
United States Supreme Court decisions, the student will be 
able to select the one winch is least consonant with previous 
decisions of the Supreme Court, 

D. The student will know the importance of a balanced diet. 

E. After hearing a lawyer's analysis of five case studies 
involving torts, the student will comprehend the principles 
of torts in each of the next five cases. 
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2. If the following objective deserves to be re-written, do so. 

Objective: Students in college will be sensitive to the 

importance of physical exercise. 

3. What is meant by the “measurability” of objectives? 

4. What is the significance of the word “domain for educational 
objective writers? 

5. List the subdivisions of the cognitive domain of educational 
objectives. 

* ... 

6. List your objections to using objectives. 

7. Without referring tc the content of this chapter, write three 
1 CO-word paragraphs defending the widespread use of 
measurable objectives. 

The reader is advised to complete the exercises in the working unit 
in combination with a colleague, or two, or three. Check answers. 

Discuss differences. 
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COURSE OBJECTIVES 


Survey of Intelligence Iiiformation Systems 

1. To familiarize the student with the capabilities, limitations, contents, 
and applicability of several major operational intelligence systems in the 
Intelligence Community. 

2. To introduce the student to developments in Systems Concepts as a 
vehicle for the cross-fertilization of techniques and ideas throughout 
the Intelligence Community. 

3. To improve the student's ability to communicate with professionals 
in this field. 

4. To provide a wider range of rational choices to the professional in 
the analysis, problem solving, and decision making tasks he encounters, 
through an introduction to Community-wide applications of Information 
Science techniques. 

Application of Information Science to Intelligence Functions 

1. To familiarize the student with the terminology and basic techniques 
of information science. 

2. To develop the student's capability to identify and define problems in 
his professional intelligence field by .using information science .techniques 
and to solve such problems at the elementary level. 

3. To improve the student's communications capabilities in conferring with 
information science professionals on more difficult problems and requirement 

4. To encourage the student to pursue the development of his own, and 
his agency's information science resources and capabilities . 

5. To provide a wiier range of rational choices :o the professional in the 
analysis, problem solving, and decision making tasks of his functional area 
of responsibility through an introduction to the oasic tools and language 
of information science. 

Information Science for Manag ers 

1. To familiarize the student with the terminology and basic techniques 

of Information Science. ' ( 

2. To identify the capabilities, limitations, and applicability of 
Information Science methods to the functions of management and the 
processes of Intelligence. 

3. To develop the student's ability to recognize applications of systematic 
methods of analysis in his professional intelligence field, and to apply 
these methods to his work. 
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Information Science for Managers (cont.) 

4. To improve the student’s ability to communicate and collaborate with 
systems professionals on more difficult problems and requirements. 

5. To provide a wider range of rational choices to the manager and his 
staff in the analysis of problems in the decision-making tasks of his 
organization. 

Operational Records I 

I STAT 


Operational Records II 

I I STAT 


Operational Records II I 

STAT 


Operational Records and Desk Orient a tion (for CTs ) 

There is no specific statement of objectives for this course. 
However, the objectives can be discerned fairly readily by persual 
of the two-page "syllabus". 

Clan de stin e Operations Orientation for DD/S&T 

To acquaint analysts from the DD/S&T and officers from other 
Agency components with the missions, organization, and concept of 
operations of the Clandestine Service. The aim of the orientation 
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Admin i strative Procedu res 

Upon completion of the first part of the course students will be able to: 

1. Describe the origins of Central Intelligence Agency and its role 
in the United States Government. 

2. Describe the organizational relationships of Agency components. 

3. Identify major tasks of the DDO as authorized by National Security 
Council directives. 

4. Use words and terms from the Glossary of Operational Terminology. 

5. Define code designations: pseudonyms, cryptonyms, aliases and 
indicators. 

6. Prepare and route correspondence for the field in the prescribed 
format. 

7. Prepare claims for domestic temporary duty travel’ and miscellaneous 
expenditures. 

DDO Students will remain an additional day. and learn to: 


STAT 
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Admlnlstra tive Pro ce dures ( c on t . ) 

the Division or Staff Records Management Office, and the Information 
Services Group. 

11. Name non-DDO offices of the Agency and other government offices or 
agencies that maintain repositories of biographic information available 
for name, checking. 

12. Select the appropriate forma to use in opening official files and in 
requesting name checks and approvals for use of agents. 

13. Identify types of DD0 project action papers. 

14. Prepare a project action routing sheet. 

Advanced Management Pro gram 

To give all new middle managers (generally GS-13 - 15 who have just 
become supervisors of supervisors) a better understanding of the major 
areas of management knowledge as a necessary step toward improving 
managerial performance in the Agency. The program will not repeat 
material covered in the Fundamentals of Supervision and Management (FSM) . 

It is assumed that the student has either taken che FSM or can if needed. 

Further, the Program is not aimed at interpersonal relations and team 
buildirng. It is assumed that the student will look to the Managerial 
Grid Seminar for that type of training. 

Field Adm inis tration 

I STAT 


Fundamentals of Budgeting 


1. Instruction and practice in the techniques used in developing the dollar 
resource requirements phase of budget formulation. 

2, Instruction and practice in the mechanization of processing budget data. 
Per form anc e App raisal Wor kshop 

1. The importance of PA and its varying functions. 
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Perfo rmance Appraisal Workshop (cont.) 

2. Specific policies of the Agency and the Office of Logistics in regard 
to PA. 

3. The function of the FR narrative and types of items to Include and 
exclude. 

4. The objectives of the PA interview, the subjects which should be 
dealt with, some alternative approaches to conducting it and specifically 
the importance of being a good listener. 

5. Several approaches to increase objectivity and decrease subjectivity 
in perparing FRs. 

6. The relationship of the PA to the entire supervisory process. 
Management By Objectives Workshop 

No data 

Fun damentals of Supervision and Management 

The overall course objective is to modify student on-the-job behavior 
in ways which will enhance his effectiveness as a supervisor and/or 
manager. Specifically, such changes will relate to communications, 
decision-making, problem-solving, motivation, perception, team action, 
leadership /manage rial styles, and managerial responsibilities. 

Managerial Grid 

1. Learn the Grid as a framework of thought. 

2. Gain insight into own Grid styles. 

3. Increase personal objectivity in work belia\ior. 

4. Reexamine managerial values. 

5. Test ways to increase effectiveness. 

6. Study use of critique. , ‘ 

7. Develop standards for openness and candor. 

8. Examine the need for active listening. 

9. Study barriers between teams. 

10. Study conflict within teams. 

11. Understanding impact of work culture on behavior. 
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Managerial Grid (cont.) 

1.2. Gain appreciation of Grid OD and outcomes possible. 

Office Man a gement Cou rse. 

1. Understanding and applying the principles of effective office 
management . 

2. Developing and employing better communication and motivation skills. 

3. Developing positive attitudes to increase her ability as an assistant 
to an executive. 

4. Improving her interpersonal relationships with her immediate 
supervisor as well as with other employees. 

The Project Officer i n the Contract Cycle 

1. To provide an understanding of the Agency’s contracting process, 
particularly as it applies to research, development, and engineering. 

2. To provide an understanding of the management responsibilities of 
the Project Officer during the contract cycle. 

Clerica l Or ientatio n 

To introduce the new employees to basic, jot -related information 
about the Agency. At the end of the training they will be able to, for 
example, explain the Agency's organization and functions; identify the 
top executives; use the multi-button telephones; open and secure safes; 
and prepare simple correspondence (memoranda, dispatches, and cables). 

Effective B riefing 

This course provides an opportunity for part icipants to learn proper 
briefing techniques through the conducting of briefing sessions on their 
own. In addition to receiving constructive criticism from the instructor 
and their colleagues, all participants will be given the opportunity to 
criticize themselves objectively by means of video tape playback sessions 

Effective Writin g < ’ 

This course concentrates on such fundamentals of writing as planning, 
organization, outlining, sentence structure, subject/verb agreement, and 
parallelism. 
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j. n . forma t ion- Reporting, Reports and Requirements 

To provide the Directorate of Operations officer with the specialized 
competence necessary to ensure that information, responsive to require- 
ments, reaches the consumer quickly, in clear language and in its most 
useful form. 

Information Reports Familiarization 

To familiarize DDO employees with the basic principles of organization, 
expression, style and format involved in the preparation of regular and 
cable information reports. The course covers official policies and 
procedures for preparing intelligence reports. It also includes instructor 
directed practice in the preparation of information reports. 

Intelligence Production Course 


To help the DDI professional become more knowledgeable on the purposes, 
methods, and problems of intelligence production in the Directorate of 
Intelligence and to appreciate better his own role in it. 

Intelligence Research Techniqu es 

This course is designed to meet the needs of Photo Interpreters, 
Research Analysts, Editors and Intelligence Assistants in NPIC. Its 
objectives are to familiarize these professionals with those offices 
of CIA and other intelligence agencies which can provide them with the 
information they need, and to provide lines of communication for 
obtaining tbe information. These objectives are achieved through 
lectures, brief ingss, and visits to libraries, dissemination centers, 
and research offices, including some visits with analysts who work on 
subjects and areas of concern to the NPIC student.. Students are en- 
couraged to determine the capability of these offices for providing 
information on their area and subject interests. 

Intelligence Writing Workshop 


The workshop examines the principles of effective intelligence 
writing, with emphasis on the analytical process in intelligence and 
problems associated with it. Aside from the general areas of clarity, 
accuracy, logic, and structure, the course .deals with report organiza- 
tion, writing techniques used in the Agency's intelligence production 
offices, and actual writing assignments based on intelligence informa- 
tion from these offices. Each session begins with a lecture covering 
the topic under discussion. During the balance of the session, students 
work on assigned exercises and reports writing projects. At that time, 
they are also given extensive individual counseling by the instructor, 
who reviews their work in detail. 
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Intelligence Writing Technique s 

1. To improve the student's understanding of the principles of 
effective intelligence writing. 

2. To provide him with a better grasp of the analytical process in 
intelligence writing and of the problems associated with these processes. 

3. To help the student master such aspects of intelligence writing as 
clarity, accuracy, logic, and structure. 

4. To help him learn report organization and writing techniques used in 
the offices and directorates of CIA. 
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COURSE OBJECTIVES 


Survey of I ntelli gence Info rma tion Systems 

1. To familiarize the student with the capabilities, limitations, contents, 
and applicability of several major operational intelligence systems in the 
Intelligence Community. 

2. To introduce the student to developments in Systems Concepts as a 
.vehicle for the cross-fertilization of techniques and ideas throughout 

the Intelligence Community. 

3. To improve the student's ability to communicate with professionals 
in this field. 

4. To provide a wider range of rational choices to the professional in 
the analysis, problem solving, and decision making tasks he encounters, 
through an introduction to Community-wide applications of Information 
Science techniques. 

Application of Information Science to Intelligence Functions 

1. To familiarize the student with the terminology and basic techniques 
of information science. 

2. To develop the student's capability to identify and define problems in 
his professional intelligence field by using information science techniques 
and to solve such problems at the elementary level. 

3. To improve the student's communications capabilities in conferring with 
information science professionals on mere difficult problems and requirements. 

4. To encourage the student to pursue the development of his own, and 
his agency's information science resources and capabilities. 

5. To provide a wider range of rational choices to the professional in the 
analysis, problem solving, and decision making tasks of his functional area 
of responsibility through an introduction to the basic tools and language 
of Information science. 


Inform ation Scienc e for Managers 

1. To familiarize the student with the terminology and basic techniques 
of Information Science. 

2. To identify the capabilities, limitations, and applicability of 
Information Science methods to the functions of management and the 
processes of Intelligence. 

3. To develop the student's ability to recognize applications of systematic 
methods of analysis in his professional intelligence field, and to apply 
these methods to his work. 
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Information Science for Managers ( c on t . ) 

4, To improve the student’s ability to communicate and collaborate with 
systems professionals on more difficult problems and requirements. 


5. To provide a wider range of rational choices to the manager and his 
staff in the analysis of problems in the decision-making tasks of his 
organization. 


Ope rati onal Records I 


STAT 


Oper atio nal Records II 


STAT 


Operational Records II I 


■ 

Opera ti onal Records and Desk Orientation (for CTs ) 


There is no specific statement of objectives for this course. 
However, the objectives can be discerned fairly readily by persual 
of the two-page "syllabus". 


Clandestine Operations Orientation for DD/S&T 


To acquaint analysts from the DD/S&T and officers from Other- 
Agency components with the missions, organization, and concept of 
operations of the Clandestine Service. The aim of the orientation 
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Adminis tra tive Proced u res 

Upon completion of the first part of the course students will be able to: 

1. Describe the origins of Central Intelligence Agency and its role 
in the United States Government. 

2. Describe the organizational relationships of Agency components. 

3. Identify rnajoi tasks of the DD0 as authorized by National Security 
Council directives . 

4. Use words and terms from the Glossary of Operational Terminology. 

3. Define code designations: pseudonyms, cryptonyms, aliases and 
indicators. 

(S. Prepare and route correspondence for the field in the prescribed 
format. 

7. Prepare claims for domestic temporary duty travel and miscellaneous 
exp end x tin res * 

STAT 

DDO Students will remain an additional day and learn to: 
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Admin :L st ra t ive Procedures (cont.) 

"" — sw 


Adva nced Management Pr ogram 

To give all new middle managers (generally GS-13 - 15 who have just 
become supervisors of supervisors) a better understanding of the major 
areas of raanagemenl: knowledge as a necessary step toward improving 
managerial performance in the Agency. The program will not repeat 
material covered in the Fundamentals of Supervision and Management (FSM) • 

It is assumed that the student has either taken the FSM or can if needed. 

Further, the Progr am is not aimed at interpersonal relations and team 
building,, It is assumed that the student will look to the Managerial 
Grid Seminar for t rat type of training. 

STAT 

Field Admin istration 


Fu nd am en tals of Bud geting 

1* Instruction and practice in the techniques used in developing the dollar 
resource requirements phase of budget f orxnulation. 

2* Instruction and practice in the mechanization of processing budget data. 
Performance Appr ‘aisa l W ork shop 

I, The importance of PA and its varying functions. 
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Perfo rman ce Apprai s al Workshop (cont . ) 

2. Specific policies of the Agency and the Office of Logistics in regard 
to PA. 

3. The function of the FR narrative and types of items to include and 
exclude. 

' 4. The objectives of the PA interview, the subjects which should be 

dealt with, some alternative approaches to conducting it and specifically 
the importance of being a good listener. 

5. Several approaches to increase objectivity and decrease subjectivity 
in perparing FRs. 

6. The relationship of the PA to the entire supervisory process. 
Manag e ment By Obj e ctives Workshop 

Wo data 

Fundam ent als of Supervision and Management 

The overall course objective is to modify student on-the-job behavior 
in ways which will enhance his effectiveness as a supervisor and/or 
manager. Specifically, such changes will relate to communications, 
decision-making, problem-solving, motivation, perception, team action, 
leadership /managerial styles, and managerial responsibilities. 

Mana geri al Grid 

1. Learn the Grid as a framework of thought. 

2. Gain insight into own Grid styles. 

3. Increase personal objectivity in work behavior. 

4. Reexamine managerial values. 

5. Test ways to increase effectiveness. 

6. Study use of critique. 

7. Develop standards for openness and candor. 

8. Examine the need for active listening. 

9. Study barriers between teams. 

10. Study conflict within teams. 

11. Understanding impact of work culture on behavior. 
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Managerial Grid (cont.) 

12. Gain appreciation of Grid OD and outcomes possible. 

Offic e Management Cotirse 

1. Understanding and applying the principles of effective office 
management . 

2 . Developing and employing better communication and motivation skills. 

3. Developing positive attitudes to increase her ability as an assistant 
to an executive. 

4. Improving her interpersonal relationships with her immediate 
supervisor as well as with other employees. 

The. Project Officer in the Contract Cycle 

1. To provide an understanding of the Agency’s contracting process, 
particularly as it applies to research, development, and engineering. 

2. To provide an understanding of the management responsibilities of 
the Project Officer during the contract cycle. 

Clerical Orienta ti on 

To introduce the new employees to basic, jcb-related information 
about the Agency. At the end of the training they will be able to, for 
example, explain the Agency's organization and functions; identify the 
top executives; use the multi— button telephones; open and secure safes, 
and prepare simple correspondence (memoranda, dispatches, and cables). 

Effective Brief ing 

This course provides an opportunity for participants to learn proper 
briefing techniques through the conducting of briefing sessions on their 
own. In addition to receiving constructive criticism from the instructor 
and their colleagues, all participants will be given the opportunity to 
criticize themselves objectively by means of video tape playback sessions 

Effective Writing 

This course concentrates on such fundamentals of writing as planning, 
organization, outlining, sentence structure, subject/verb agreement, and 
parallelism. 
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Intelligence Product ion C ourse 

To help the DDI professional become more knowledgeable on the purposes, 
methods, and problems of intelligence production in the Directorate of 
Intelligence and to appreciate better his own role in it. 

Intel li gence Research T echn iques 

This course is designed to meet the needs of Photo Interpreters, 
Research Analysts, Editors and Intelligence Assistants in NPIC. Its 
objectives are to familiarize these professionals with those offices 
of CIA and other intelligence agencies which can provide them with the 
information they need, and to provide lines of communication for 
obtaining the information. These objectives are achieved through 
lectures, briefings, and visits to libraries, dissemination centers, 
and research offices, including some visits with analysts who work on 
subjects and areas of concern to the NPIC student;. Students are en- 
couraged to determine the capability of these offices for providing 
Information on their area and subject interests. 

Intelligence Writing Workshop 

The workshop examines the principles of effective intelligence 
writing, with emphasis on the analytical process in intelligence and 
problems associated with it. Aside from the general areas of clarity, 
accuracy, logic, and structure, the course deals with report organiza- 
tion, writing techniques used in the Agency’s intelligence production 
offices, and actual writing assignments based on intelligence informa- 
tion from these offices. Each session begins with a lecture covering 
the topic under discussion. During the balance of the session, students 
work on assigned exercises and reports writing projects. At that time, 
they are also given extensive individual counseling by the instructor, 
who reviews their work in detail. 
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Xntelll gence Writ i n g Technique s 

1. To improve the student's understanding of the principles of 
effective intelligence writing. 

2. To provide him with a better grasp of the analytical process in 
intelligence writing and of the problems associated with these processes. 

3. To help the student master such aspects of intelligence writing as 
clarity, accuracy, logic, and structure. 

4. To help him learn report organization and writing techniques used in 
the offices and directorates of CIA. 
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AIS COURSE OBJECTIVES 


I. To Review: 

- CIA as a Total Organization 

- Across -the-Board Problems of the Agency and 

the Intelligence Community 

- The Agency's Relationships to the NSC and White 

House 


II. To Consider and Evaluate: 

- Significant International... Developments and 
Their Effect on U.S. Foreign Policy 


III. To Assess Important Domestic Trends of Relevance 
to the Agency 


IV. In general, to encourage a frank exchange of views 
thrDughout Seminar and an enhancement of each 
member's professional contribution to., the 
Agency. 
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OBJECTIVES OF T HI- CIA SENIOR SEMINAR 

y 

1. The requirement for an Agency training program depends- 

on whether the participants can benefit in one or several important 
dimensions tvith respect to current or future Agency jobs which a 
properly designed course can provide. Training of nettf officers 
‘is primarily concerned with equipping them with intelligence and 
related job skills and with orienting them to the Agency and to 

■ 'i 

the intelligence business. In contrast, the senior officer largely 
possesses, by virtue of his advancement through the ranks, the sub- 
stantive knowledge and skills related' fo his job. In this sense, 
the senior officer must be regarded as the expert insofar as the 
execution of his j ob- is concerned, and a training program cannot 
add significantly to the skills he uses directly in his work-. 

2. There is, of course, much more to being an effective senior 
officer in CIA than mastery over the 'content and skills needed for 
a particular assignment. For the senior officer to share effect- 
ively in the management of the Agency, he must have some insight 

into the external pressures, internal problems, and other factors 

¥ 

bearing on the decisions of top management. He should have a gra;p 
of important Agency business and significant Agency relationships 
with the rest of government and outside groups which fall outside 
the immediate purview of his job. Such knowledge not only bears 
on his own motivation and understanding of the Agency, but is a 
necessary ingredient in his ability to interpret to his subordinates 
the decisions and actions of top management in such a way as to 


Approved For Release ^OOJifVI/Qt : 


CIA-RDP78-0621 5A0001 0001 0007-0 



. Approved For Release 2002/11/01 : CIA-RDP78-06215Afr001 0001 0007-0 

- 2 - 

avoid parochialism. In the evolution of the Agency, security 
requirements and our tradition of compartmcntation have tended 
to carry over into this area and to limit many senior officers 1 
understanding of relevant organizational matters. In the process 
of developing the Senior Seminar, a number of top Agency figures 
noted that most of our senior officers are wanting in their over- 
• all understanding of Agency management. This is a serious de- 
ficiency inasmuch as the senior officer group must be prepared to 
take on greater responsibilities over ' .the next decade in the 
running of the Agency. Meeting this requirement identifies the 
first objective of the Senior Seminar. 

Objective 1 . To enable .Seminar participants to develop 
greater insight into the problems and pressures facing CIA 
management, the processes of change within thfe Agency and it 
external relationships, and- developments in American society 
which are relevant to CIA as an organization. 

3. A statistical analysis of those senior officers who may 
be candidates for tire Senior Seminar indicates that most of this 
group entered the Agency in the early 1950s and completed their 
academic training approximately two decades ago. During this 
lapse of time, many changes have occurred in foreign countries 
and regions, in U.S. foreign policy and in domestic factors beari 
on policy, as well as in interpretive thinking on subjects relate 
to intelligence and foreign policy. There is a need to update 

the knowledge of many senior officers on these areas of change. 
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Obj cctivc" 2. To acquaint Seminar, participants with current 
thinking on U. S. foreign policy equities and developments 
abroad which arc the intelligence collection, analysis and 
covert action targets of CIA. 

4. The occupational picture which emerges from the statistical 
analysis is that of a group of officers who may have become ac- 
customed to the pattern and regimen,. of Agency employment. In the 
process, perceptions and sensitivities toward work colleagues -- 
peers as well as subordinates - -have in- varying degrees become 
dulled. Many senior supervisors cite difficulties in understanding 
and relating to younger . of ficers . The Seminar provides an oppor- 
tunity to examine this area. 

Objective 3. To provide an opportunity for Seminar officer 
to refresh and broaden their understanding and appreciation 
of ’’the other guy." 
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LATIN AMERICAN AREA SEMINAR 
OBJECTIVES 


The Latin American Area Seminar is designed 
to provide Agency officers with both a broader 
perspective on area problems and a deeper insight 
into how such problems may affect U.S. interests 
and future requirements for intelligence on the 
area. It is organized to enable participants to 
examine Latin American problems from a variety of 
viewpoints , including those of Latin Americans, 
academicians, and U.S. intelligence analysts. 
Reading materials, lectures, and panel discussions 
are designed to stimulate participation by members 
and to facilitate sharing of experience, s and ideas. 
In addition, each member will be asked to prepare 
a report on a particular area or country problem 
for presentation during the final sessions of the 
seminar. 
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CHINA FAMILIARIZATION COURSE 
OBJECTIVES 


This course is designed to assist the intelligence 
fficer working on China matters: Its main aims: 

1. To provide information on the fundamental 
geographic, political, and cultural features of 
China and the Chinese people and on the nature 

of China's power position, 

2. To introduce the student to the nature of 
the Chinese language and certain methods used in 
handling the written language., , 

3. To acquaint him with sources .of informa- 
tion on China. 


9 
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USSR COUNTRY SURVEY COURSE 
OBJECTIVES 


The fund oriental objective is to provide a comprehensive 
understanding of the Soviet Union appropriate for both the 
analyst and the operations officer. More specifically, the 
course seeks to describe and analyze the geographic intel- 
lectual and institutional environment within which: 

1. decisions on political, economic, foreign 

policy, and other concerns, are made and 
executed; •: 0 

2. the daily experiences and needs- of the 
Soviet citizen arc determined; 

3. the potential for growth (economic, mili- 
tary, technological) as a national power 
may be estimated. 

The course responds to the needs of diverse professionals 
including the analyst or operations officer new' to the Soviet 
field and those persons who have been ' focus ing on a narrow 
Soviet target but; desire broader ‘perspectives. 


Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 0007-0 



Approved For Retease 2002/11/01 : CIA-RDP78-06215A^01 0001 0007-0 

•SUPPORT SERVICES REVIEW: TRENDS & HIGHLIGHTS 

OBJECTIVES 

1. To provide a training framework in vhich personnel of the 
Support Services will learn of significant programs and 

activities of the various Support Offices and will learn 

- ■} 

of policies and trends in M&S . 

2. To provide an environment in vhich individuals from various 
offices of Support Services will get to Enow one another 

in order to enhance future work relationships. 
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JCS-DIA OREINTATION 

OBJECTIVE 

To introduce the class members to key officials 
of the CIA, who x^ill discuss the organization and 
responsibilities of the Agency's major components, 
and to cover the Agency's interrelationships with 
the other agencies and departments in the Intelligence 
Community. Jti 
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1. To inculcate pride in tke professionalism of tlie 
' .Agency anti its work. 


2. To widen perspectives Leyond one s own jok 
and experience. 

■> 

3. To create team spirit flirougk greater knowledge 
of ike missions, functions, personnel and problems 
in oilier offices. 

TT o provide greater knowledge of orker Community 
agencies. 

5 . To provide greater knowledge of internati onal 

affairs and domestic matters as iliey pertain to 
foreign relations. 

S. 1 o 3 >r ovi d e nnv insiglifs into managerial styles 
and proklems. 


7. JP o i* cm oil v at e mid-grade officers for taking on 

new responsibilities. 
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ORIENTATION FOR OVERSEAS 


The objectives of this course ere; 

(a) to orient you to some of the important environmental end 
psychological challenges which are commonly met in living and 
working overseas; 

(b) to provide practical advice on how to respond to these challenges. 


Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 0007-0 



Approved For Rrfease 2002/11/01 : CIA-RDP78-06215A%Sbl00010007-(j 

\ 


3KTEILIC-EUCE HI WORLD AFFAIRS 
Course Objectives 


Ihis course, designed for the nev CIA officer, is 
intended to: ■ 

i 

3_. Provide a basic understanding of CIA, its or- 
ganization and functions, and a general gra.s.p 
of the Agency's relationships ■with other parts 
of the US Intelligence Community. 


2. Introduce the fundamentals of intelligence 
end the relationship of the intelligence 
process to US foreign policy. 


3 . Review important operational and analytical 
factors and problems of intelligence concern 
in key stress of the “work, "with ns jcr.a oten- 
tion on the USS’ ? and China ana brieier con- 
sideration of Jsivan and Europe and problem 
areas of Latin .America, Asia and Africa, 
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Memorandum 


0IRr8846 


TO : Director, OTR 


DATE: 20 July 1973 


from : 


SUBJECT: Program Performance Measur em ent . 

The Emerging Role of OTR in establishing 
Requirements for Training. 

1 am submitting this memorandum intended as a discussion of 

(1) how to provide the data base necessary to tell OTR how well 
the programs are meeting their objectives on a current basis 

(2) an approach to determine to what extent the knowledge and/or 
skills imparted are applied on the job and 

| (3) h° w can establish requirements as contrasted with 
I. historically reacting to requirements. 

A concept of a data base is presented which is intended to aggregate 
all of this and also attached is a basic form that can be used to 
respond to all of these goals. 

In order to institute the best type of (program) performance 
measurement a numoer of aspects must be considered which not only influence 
the behavior and norale of those being measured but also affect the ability 
to satisfy the student — our customer. 

This paper is divided into four sections 

Section 1 - A Discussion of Program Performance Measurement 
Section 2 - OTR’s Role in Establishing Requirements 
Section 3 - Some Observations 
Section 4 — Appendix A - C 
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A Discussion of Program Performance Measurement 

There are four steps involved in Program Performance Measurement 
that must be followed in a continually repetitive sequence to achieve 
the full benefits of such efforts. 

The first step requires the careful specification and analysis of 
basic program objectives in each major area of activity. To accomplish 
this one must back away from the particular program being carried on, 
look at their objectives, and ask what are we really trying to 
accomplish. This definition of goals to be accomplished should start 
ideally at the top of the "organization" so that each level can be 
certain that their definition of goals falls within the scope of what 
has been defined by the next higher level. OTR certainly must 
provide the training necessary for agency personnel to accomplish or 
improve the accomplishment of the various agency missions but such 
definition is too broad and must be narrowed. At the other extreme 
for OTR to say we shall train "x" number of people or increase our 
student through-put by some quantity is too narrow and must be 
broadened. Specification of OTR objectives must fall between these 
two extremes. The more we learn about how to reach an objective, 
the more clearly we understand the objective resulting in a constant 
interaction between the decision process and our knowledge of our 
true objectives. 

The second step involves the analysis of the output of a given 
program in terms of the objectives initially specified in the first 
step. For the OTR programs, our output addresses not how many or 
how many more students did we turn out but rather what improvements, 

• — 2 — 
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knowledge and/or new skills have we provided to the student. Will 
our educational efforts influence his behavior on the job and improve 
his job performance. This is often referred to as feedback a term 
taken from engineering servomechanism theory. 

The third step calls for measurement of the total costs of the 
program not just for one year but over at least several years 
ahead. This would require OTR to identify the resources expended 
in each program, to accumulate resource costs by program and to 
extrapolate into the future periods from a historical cost data 
base. 

The fourth step involves the analysis of alternatives if and 
only if programs are competing for limited resources. As an 
expository application let us examine how these steps or this cycle 
of events applies to the Information Science Program and the 
Information Science for Intelligence Functions Course wit hin that 
program. 

Course objectives serve two purposes: 

1. They express the desired results of our customers in 
terns of accomplishments or goals to be achieved by 
the course. 

2. They provide the basis or elements fundamental to course 
evaluation. 

The most difficult aspect of any professional performance 
measurement scheme is not how to measure but what should be measured. 
What should be measured is dependent upon how one is organized and 
whether accountability and responsibility for activities which 
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f 

satisfy customers needs are clearly defined and understood by those 
charged with such responsibility and accountability. Clearly 
organizational analysis is beyond the scope of this paper nevertheless 
the interrelationship between the design of an organization and an 
evaluation of an organization's performance must be clearly established. 

These ideas have been used in our Information Science Program. 

r 

For example I have assumed from our previous planning that the single 
overall objective required by step one of our Information Science 
Training Program at the Chief /Information Science Training Staff level 
is to educate Intelligence Professionals in the Information Science 
Disciplines. For the Current FY 73 Community Program segment, the 
numbers of students planned were as follows: 

2 offerings — Survey — 3 weeks — 30 stdts = 60 

2 offerings — Functions -- 4 weeks — 25 stdts = 50 

2 offerings — Inf. Sci. — — 1 week — 25 stdts ~ 50 

Total Students 160 

Total Course VJeeks 16 

Assuming for step two that the proper output measure of the 

Information Science Program objective is to quote numbers of students 

exposed to Information Science then I could certify for the Functions 

Course that 27 + 30 or 57 students completed the Functions Course 

during IT 73. One can further document the specific inputs (and 

their costs required in step three) required to generate this output 

of 57 students i.e., lecture time of resident and guest faculty, [ 

( 

course preparation time, computer time and charges (by problem and f 

lecture if necessary) etc. From a Systems Aitalysis point of view, 

V 

- 4 - ‘ { 
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the output (57 students) is tied to the related and required inputs 
(the resources and their costs) but is this a sufficient and proper 
measure? This measure of output as a single measure reflects the 
number of students put through the course but does not reflect the 
full range of values or benefits provided to our students and their 
respective organizations. In order to measure the value provided 
by the Functions Course and its contribution to the Information 
Science Program overall, it is necessary to go beyond the single 
measure of student output and resource input to determine whether 
we accomplished the course objectives and accordingly satisfied 
the students our customers. What is missing is the careful specification 
and analysis of the program objectives which produces a more meaningful 
output measure. 

To elaborate further — for the Functions Course alone — four 
priority ranked specific student-oriented objectives were established 
(step one) : 

1. To familiarize you with the terminology and basic 
techniques of Information Science. 

2. To develop your capability of identifying and defining 
problems in your professional intelligence field which 
are amenable to solution by information science techniques 
and to solve such problems at the elementary level. 

3. To improve your comnfunications capabilities in conferring 
with information science professionals. 

4. To encourage you to pursue further the development of 
your own, and your organization's information science 
resources and capabilities. 

How well we accomplish these objectives in the Functions Course 
with a fixed set of resource inputs is reflected in the course 
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evaluations completed by the students. A summary of the student 
evaluation responses from the last Functions Course demonstrates 
to what degree these objectives were fulfilled for that particular 
class (step two) : 

STUDENT EVALUATION-FINDINGS 

OBJECTIVE OUTSTANDING/EXCELLENT GOOD FAIR 


1 . 

27 Students 

3 

0 


90% of the class 

10% 

— 

2. 

15 

13 

2 


50% 

43% 

7% 

3. 

21 

7 

2 


70% 

. 23% 

7% 

4. 

23 

6 

1 


77% 

20% 

3% 

From these 

findings one can conclude that 

all objectives 

were 


accomplished. Since objectives 2 and 3 rank the lowest, efforts 
should be directed toward improving those- areas for the next 
offering of the Functions Course. A follow-up questionnaire, four 
to six months after the course, would further confirm the degree of 
job application rccomplished by the students of each course. 

These objectives and the associated evaluation scheme used for 
the Functions Course may not be applicable for all courses conducted 
by 0TR. Each course must have specified its own set of objectives 
based on the unique requirements of the customer needs to be satisfied 
by each course. How each course is conducted depends upon (1) the 
subject material to be presented, (2) the skill and knowledge of the 
assigned manpower presenting the course and (3) any procedural 

directives issued by top and/or middle management that apply 
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specifically to a particular program or course. These are the 
factors that influence not only the specification of objectives 
but also the degree to which the course objectives can be 
accomplished . 

OTR's Role in Establishing Requirements 

Although this process, its steps and their interrelationship 
have been explained using the Functions Course as a "real live 
intelligence application" the same concepts apply to any level within 
the organization. For example, the objectives that OTR must accomplish 
have in the past depended upon the requirements that have been 
levied by "Top" management. How OTR is organized has depended upon 
(1) the entire list of such requirements and (2) the aggregation of 
similiar activities and/or courses (requirements) into 'manageable 
(school) segments. Such aggregations must provide for clearly defined 
areas of responsibility and authority which are the pre-requisites to 
accountability. To say the least I have been disappointed in the limited 
agency career development program that exists only for specific segments 
or very narrow specialties. The time has come for OTR to become more 
aggressive not only in career development training but also in much 
of the currently named "component" training. The training that OTR 
conducts for the Directorate of Operation^ and the Directorate of 
Intelligence I consider to be basically component training. In order 
to effect batter utilization of training assets by OTR, OTR must 
effectively plan, organize and control the total training effort of 

- 7 - 
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the agency. Planning sets the stage, organizing sets the resources 
and controlling sets the degree to which the organization will 
continue to survive. Planning determines a schedule of what is to 
be accomplished. Organizing determines the resources to be invested 
and how to accomplish the planned schedule and controlling determines 
how well the plan was accomplished. Since control requires a 
measured comparison of what was accomplished to what was planned to 
be accomplished, there can be no control without a succinct plan. 

The degree or amount of control required is dependent upon the 
professional capabilities of the personnel. In this regard OTR must 
decide what role it is to play in Agency Training. This really means 
a restatement of the objectives to be accomplished, time-phased in a 
reasonable balance between assets, resources and time for accomplishment. 
I am reminded of the little boy who was asked by his father (a world 
reknowned Production Control expert) what he wanted for his birthday 
next month. The boy replied "a baby brother." His dad responded 
with "that’s impossible son." To which the boy quickly replied 
"you have taught me all of the principles and practices of effective 
production control, just put more men on the job". 

Once the objectives are carefully specified and analyzed, then 
OTR can determine if the present organization is properly structured 
to accomplish this priority-ranked list of objectives. The only 
definitive principles of organization structure that apply are that 
(1) each organization falls somewhere between the extremes of being 
functionally organized and being product or service organized and 

* 

- 8 » 

Approved For Release 2002/11/01 : CIA-RDP78-06215A0001 0001 0007-0 



Approved For Rftfease 2002/11/01 : CIA-RDP78-O6215AO0O1 0001 0007-0 

(2) like-activities should be grouped into manageable segments so 

that the specified objectives of such "grouped" activities can be 
responsibly accomplished. 

If OTR is to exert more influence in career development, then 
OTR must write career development plans (programs) in conjunction 
with each responsible directorate. These career development programs 
then become the focal point of the OTR program planning and the 
subsequent course planning. The existence of career development 
plans does not require OTR to be organized along the same career 
structure. In fact, an OTR element such as the Information Science 
.Program can cut across a number of career development plans by 
providing segments of instruction or courses that fit into various 
career progressioi patterns. Other such examples include the IWA 

introductory orientation, the mid-career and management courses and 
the senior seminar. 

If OTR recognizes a need or void in training then a course or 
program should be developed and presented. Demand for such a program 
can be generated through effective advertising campaigns. We should 
look to the methods used in successful marketing research and advertising 

campaigns and apply such tactics to researching the need and promoting 
newly developed programs. 

To determine how Bell each course Is meeting Its objectives, I 
offer as an example (appendix A) the form used In the Functions Course. 

As a follow-up questionnaire, I offer the form (appendix B) designed for 
the Functions Course. To determine the allocation of manpower/tlmo 

assets, I offer the Faculty/Staff Activity Report (appendix c) as the 
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basic time allocation data collection device. One can thus relate 
resource expenditures to the benefits provided to our customers — 
the students and to those responsible for review of career development. 

Some Observations 

It is often charged that such measurement sets up biases in 
decision-making by concentrating on costs and ignoring intangibles 
and human factors which cannot be quantified. Or conversely by 
naively attempting to put numbers on such imponderable elements 
thereby misleading the decision-maker . Such thinking often forces 
personnel to play games. For example (in the evaluation of personnel 
who (1) present lectures or (2) participate as guest lecturers in 
other courses.) If I knew that my performance was to be evaluated 
on (1) the number of students who were lectured (2) the number of 
my lecture hours and (3) the ratio of students per lecture hours 
then I would make certain that my lecture hours per course were at 
a ma ximum and that I lectured only to large student-groups or classes. 

I could so maximize my performance measures that I would always be 

number one on this performance roster. 

However sincere these critics may be — they reflect a complete 
misunderstanding of the relevent issue. -And sometimes they simply 
reflect the chagrin that particular pet projects may not show up 
well under such measurement schemes. 

Program Performance measurement does require a systematic analysis 
(the means) of program proposals and decisions, concentrating on those 
particular decisions (the ends) which have inherent budgetary 
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consequences. Please note that systematic analysis does not have to 
be quantitative and is not co-extensive with quantitative analysis. 

The word analyze" does not have the same meaning as the words 
"quantify" or "measure" although analysis often includes some form 
of measurement. Management by Objectives, the PPB process or whatever 
the current "in" title of a good management philosopy is, all of these 
concepts seek to subject to a systematic analysis both the tangible 
and intangible elements of a program decision. We live in a world 
that must make decisions often using limited or meagre inf ormation. 
This is more akin to the European Style of Management which opts 
for a decision as contrasted with the American Style which opts for 
"adequate" information before making the decision. In any event 
let us not become so Management by Objectives (MBO) oriented that 
we become managers who can’t make decisions unless the plan calls 
for a decision. 
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INFORMATION SCIENCE FOR INTELLIGENCE FUNCTIONS 


CLASS: 

INT RODUC TION. The Staff is concerned with the quality of its educational 
program. Your constructive comments are solicited and will be used as part 
pf the basis for improving the ability of future presentations of the 
Functions Course to meet the information science training requirements of 
user organizations. 

1. What is your overall Impression of this course? 


Outstanding Excellent Good Fair Poor 

Did we fulfill each of our course objectives for you? Indicate below: 

(a) To familiarize you with the terminology and basic techniques 
of Information Science. 

! 

Outstanding Excellent Good Fair Poor 


(lb) To develop your capability 
in your professional Intel 
solution by 
problems at 


1 n rorrx 


the el emantary 1 evel . 


of identifying and defining problems 
IlgencG field which nra amenable to 
:ience techniques and to solve such 


Outstanding Excellent Good Fair Poor 

(e) To improve your communications capabilities in conferring with 
Information science professionals. 


Outstanding Excellent Good Fair Poor 

(d) To encourage you to pursue further the development of your own, 
and your organization's information science resources and 
capabilities. 

Outstanding Excellent Good Fair Poor 

(e) Do you believe the course content is compatible with the course 

objectives? Yes No 

Discuss this compatibility and describe any changes that you 
think would increase this compatibility. 
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4. Would you recommend this course to a co-worker? 
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5. What Is your job title and what are your major tasks? 


6. What are your personal reason(s) for attending this course? 


7. Reflecting on your learning from the course what percent cf the course 
do you feel will contribute or be of use to you? Please identify by 
entering one (x) check for each line. 


a. Immediate Use 

50/o 7 OX 1 Ci5% 

b. Long-range Use 

2U% 50X 753 TOoT 


8. Please rote the course in light of your answers to Questions #4, #5, 
#6 and #7 , • 

Outstanding Excellent Good Fair Poor 


9. As a follow-up to your end-of-course evaluation » any subsequent 

comments based on work experience at your home station would be most 
welcome and helpful in updating course relevance. Would you be 
interested in accomplishing a post-graduate questionnaire four months 
after completion of this Course? 


Yes No 


3 
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3. Reflect on your learnin3v^ft^#i|ebPRg|^%et!20®2Me|«Jl. : (ff^BPTt^(^lJ^^0rt69ftt0iM)7i:§ted please indicate 
your personal reaction to the appropriate space. 


SUBJECT ELEMENT 

I can see no 
application for 
this element in 
my work. 

I am aware of 
a few appli- 
cations for this 
element. 

I am aware 
of many 
applications 
for this 
element in 
my work. 

Vd like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my 
work. 

I will certainly 
incorporate . 
the infor- 
mation and 
techniques 
from this 
element in my 
work. 

Elementary System 
Concepts 





< 

Basic Programming 

i 


; 



. 

Statistics 






Library Programs 






DELPHI 






Decision Trees 





i 

Network Analysis 






Intelligence Problems - 

making use of 
these new techniques 

i 



1 
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3. Reject on > our learnij^tom tne course this week., For each subject element listed please indicate 
your personal reaction to the appropriate space. 


SUBJECT ELEMENT 

I can see no 
application for 
this element in 
my work. 

Iam aware of 
a few appli- 
cations for this 
element. 

I am aware 
of many 
applications 
for this 
element in 
my work. 

Ld like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my 
work. 

I will certainly 
incorporate 
the infor- 
mation and . 
techniques 
from this 
element in 
mv work. 

Probability 





<— 

Linear Programming 






Basic Programming 






Single Correlation & 
Pvegression 






Library Programs 






DELPHI 






Probability Assessment 





( ' 

Intelligence Problems - 

making use of these 
new techniques 
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3. Reflect. on yonr leenni^WPiSS! please Indicate 
your personal reaction- to the appropriate space. 


SUBJECT ELEMENT 

1 . 

i can see no 
application for 
this element in 
my work. 

I am aware of 
a few appli- 
cations for this 
element. 

I am aware 
of many 
applications 
for this 
element in 
my work. 

I*d like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my . 
work. 

I will certain 
. incorporate 
the infor- 
mation and 
techniques 
from this 
element in nr 
work. 

Information Storage And 
Retrieval 





< 

Queueing 






Basic Programming 






Decision Theory 



■ — — : 



Library Programs 



1 



DELPHI 






ISS 





< 

COINS 






Intelligence Problems - 

making use of these 
new techniques 
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3* Reflect on your learning tP ? ? Jt e d please indicate 

your personal reaction to the appropriate space. 


CTT - ^ Tr'/^'T! T~'T T”' ^ •% rrn 

X^tZa 1M JL 

I can see no 
application for 
this element in 
my work. 

lam aware of 
a few appli- 
cations for this 
element. 

I am aware 
of many 
applications 
for this 

element in 
my work. 

I ? d like to incor- 
porate the infor- 
mation Sr tech- 
niques from this 
element in my 
work. 

I will certainly 
incorporate 
the infor- 
mation and 
techniques 
from this 
element in 
mv work. 

Retrieval Operations 





< 

Search Strategy 

i 





Bayesian Analysis 

“ j 




i 

1 


Semantic Distortion 






File Construction 






MIS/PP3 






Human Factors 





! 

Modelling & 

Simulation 





h _ (. 

PERT 






Intelligence Problems - 

making use of these 
n e w t c c h n i que s 
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NAME; WEEK OF: T0_ 

This fora is simple to maintain if done each day. Indicate the 
amount of time spent performing any activity listed. (Indicate 
others if necessary.) An activity is considered significant if 
it requires more than 15 minutes of your time to do it. In the 
case of short duration jobs, i.e., filing, merely record the 
approximate total time spent in that activity during the day. 
turn in to branch chief by Friday noon each week. 


ADMINISTRATION: 

Management : 

Mon. 

Tue. 

Wed. 

Thur. 

Fri. 

0.7. 

letter/report-research 
letter/ report-preparation 
letter/report-coordinating 

Brief ing s-pr epar ing 

Brief ings-presenting 

Brief ings-attending 

Meetings-preparation 

Meetings-attending 

Telephone 

Clerical: 

typing 

filing 

reproduction 

meetings 

telephone 

training 








































• 




































































— — - 



■ 





■ | 










. _ .. ' 






CURRICULUM: 

Instruction: 

Research/study 

Lesson preparation 

Lesson presentation 

Lesson attendance 

Coordinating 

































* 
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CURRICULUM ACTIVITY CONTD . 
Scheduling : 

preparation 

coordinating 

Student Advising/Evaluating 

Faculty Advisor Time 
Seminar Evaluation 
Paper/report evaluation 
Briefing evaluation 
Counseling 

Record administration 

Exercises : 

Development 
Preparation for 
Participation 


SECURITY : 

Security indoctrination/ training 
PROFESSIONALISM: 

Professional development 
Proficiency maintenance 


TRAVEL /LEAVE : 

Travel/TDY 

Leave 


1 r-eiA-RE 

Mon. 

jP-78-Oft 

Tue. 

2|45AQP&44 

Wed. 

30010007- 

Thur. 

Ci 

Fri. 

O.T. 



















-1 























r "* 1 


















. . _ — 
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Curriculum Committee Meeting DTR Conf Em 

25 October 1973 
2;00-5:00 pm 



j in the Self-Study Program; self-study means different things to 

| d i f f« ren t people. DTR wants this think thought about and talked 
1 about in this group. 


The paper presents the thing very well as to what exists at the moment and 

where the confusion exists. What are we trying to do? 

The DTR has expressed the need for the use of video cassette tapes as the 

primary teaching medium in management courses. He feels it might also be 

applied to other courses in OTR. Doing this would require an instructor ^LEGIB 

to conduct the course and handle a seminar J 

| discussions, and the problems presented, so therefor* it would not 
a 

really be/self study activity. This will require us to develop these 
cassettes and oonduct the courses, it will require equipment. The DTR 
has not clearly told us what he wants as self-study. Use of video cassettes 
in a course would cut down on the number of instructors you need, yes, but you 

still would need an instructor there. This would make the ±x cassette a 
training aid. 

Let's look at our inventory now... we have 

a. cassettes that carry a message, a specific instance... 


b. a cassette that is a course, and is indeed a self-study course, with 
a drill book, homework, and no instructors are needed. 

Does MBO have a workbook? 

It has a leader's guide to be used for discussion purposes. 

There is not, in our opinion, a 


re is not, in our opinion, a| (sufficiently sound group of ILLEGIB 
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messages to enable us to say that "here's a course." 
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Of the video cassettes available today, those helpful to us exist only in the 

fields of management and computer science; there are some available in specific 

skills training. 

Looks like a small market for us; have we surveyed the potential source? 

We have a rather complete bibliography; some are simply cassette versions of 
motion pictures. A lot of films are now showing up in the cassette series. 
Regarding management training, we have come through a history of dealing with 
package things, e.g. , the Arid, former AMP which drew much criticism, and a 
new AMP which seems to have received a favorable reaction. When we go into 
package stuff of management theory, we should look at how relevant it is to 
this Agency. If it doesn't relate we should not put too much investment in thai 
kind of program. On the other hand, if it is truly self-study where no 
instructor^ is involved, then that is a different kettle of fish. 

You won't get anything relevant to the Agency. But I thought the MB0 
series relevant . it contains a lot of things about the fundamentals of MBO. 

A person could learn a good deal. 

What about some of the training that is presently being done, say at the 
..I'm not sure this is what the DTR has in mind, but we could putSTATINTL 


±S it, e.g. something on 


on cassettes and let it be used as a 


unit of instruction and have it available for self-study. 

How much money do you have? 

Not very much, $6,000 which will only buy 3 sets of cassettes. 

We're putting in a one month rental order from this group that came to kc±f 
brief us, but they only have six subjects, only one of which applies to us. 

I propose the following: 

I think perhaps the DTR is getting ahead of the situation here with a magical 
concept which is tied into a set of facts that we can't grapple with until we 


! know more about what audio visual support and training aids we want to get 
I into. Since we are in an experimental stage, it is not to our benefit to nu 
search out lots of areas or possible extension of equipment that we don't 
^"ATINTL have much feel for at this time. | | is going to Chicago in November for an 

audio-visual seminar and we would like to learn some essential steps first 
before we start into courses. We must not let ourselves be distracted from 

to exerpiment, etc. before we can impose some applications of equipment. 
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Maybe those applications of what the DTR wants will come in due course, but at 
this moment, I feel I whould try to persuade him not to fish in these waters. 
There's a little time needed here, and we’re involved in a very experimental 
and unknown field, too. So let’s concentrate^ on Option C. 

Presently , where it is material, the video cassette system can be used in a 
course. This can just be a segment of a day’s block of instruction. 

This would not be related to self-study, it would be just a substitute for a 
film then. 

If it's part of a bloc of instruction, then it's just like a film; but if 
the equipment is there, and they use it before hours, and it's something 
extra, it could then be considered self-study. 

Why not get 6-8 pieces of equipment , put then in headquarters, advertise 
that we have them, and stand back and see what happens... 

Let’s agree that we proceed with Option C and at the same time we are 
perfectly well awareofthe DTR's position pmrt-Hm but at this time for 
reasons of a technical and financial nature, to proceed in that vein at this 
time will be taken under advisement in due course. 

* STATINTL 


Also, option D. 


STATINTL 


I d like to give a little briefing on what is going on at 
our training units, 


..on 


and Intel cables, we're putting 


STATINTL 

them 


on tape, and we hope to get to the point where the tapes are self sustaining. 
As an example, if an instructor looks at an Intel cable written by a 
student km after student has studied the tapes, he can say jfyou better 


j review tape 12 on this, it will give you the answer.* But to effect this, 

I can say it will be a long way down the pike. It could be usefal for ALT, 
and in the Ops Support andx$ AOC courses. But I can't at the moment see 

: 

it being put on the nkf shelf and having some DDO officer call for a 

particular tape. But it’s far down the pike, and it will have to be done in 

* tl Tpprove(f £ or^eleasl' stodtPRfeO® 0W*eral 
instructors and have them concentrate on getting it done. 
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! Old Business 

l _ 

i Determination of alternates: 

STATINTL f 


The Chairman reminded the membership of the creation of 


a policy Collection; asked for their contributions. 

The Cunningham paper on the training of the analyst is 
| not existent. We will not hare a paper on training for the analyst for 

STATINTL 


STATINTL 


omr 7-8 Nov mtg 


remove it from the Agenda. 


Report on meeting with 


Ire DDO counselling 


STAT 


! course 

j 

I 

We checked on who are the people they are talking about in their memo* 

j 

j This group has been put together as a career management group plus 
people in each of the divisions who serve as counsellors. 

| In dtterming what they need: they have all in one point in time supervisied 
! people and there is a desire for all of them to know what sound counselling 
i techniques are... what problems might come up in a relationship. 

I Shep’s mono is only a for instance. They would like a little bit of 

S STATINTL 

skill practice, and there is no specific deadline for us. When we mentioned 

j presentations 

| audio cassettee/on the subject, this caused some reaction from , 

| do/tro. 

! The decision was that they do need something that we can heibp with and we 

! 

j should put someone in touch with them who can help them. 

There is a guy at the Univ of Virginia who has this sort of lecture. . .exactly 
what they want. 

They would like this done internally so they can discuss their own problems. 

We suggested use of our own equipment downstaiis. Somebody ought to be 
put in touch with them and that person can follow our lead. 

We asked haw big an audience. .about 15 this time and possibly another 15. 

So it might not be a full blown course. 

We mentioned that parts of FSM might be valuable to them. 

I Wo ^pfevl8Wr'Rlte , ^e t 2o«e!TlA ! r': ffl^S^S-J&lliooolOOOIOOOy-O 

I Hew about other contributors? OMS? 

i 
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STATINTL 


sta? tatintl 

STATINTL 

STATINTL 

STATINTL 

STATINTL 


( medies ) 

Hot really necessary, we got on to this/because Shep mentioned 
transactional analysis, but remember that his wee of thewords, transactional 
analysis was only a for instance. 

This was just a gratuitious thought thrown out by Shep. 

If it is agreed that we should take it on, who will do it? 

I While functionally, it is something that would come closer to us, since 
| Art expressed an interest in it, I would be happy to defer to Art. 
j Have a full schedule now developing training prof lies .. .no time, 
j Question is: at what time can we get into business with them? 

| Seems to me that counselling falls into the area of management so give it 
| to management faculty. 

j 

| Will accept it, but would ask that Alan and Charlie give us in writing the 
| benefit of all of these discussions* Need information on some of the 

i 

j objectives so I can identify the man to assign to the Job. 
j We’ll sit down with you and whoever you nominate to handle it and talk with 


j him. 

i 

| Want paper before I select him. 

| Directed Alan and Charlie to write a paper which will give us a very good 


documentation... tell Shep andGordon that we have talked about it and we'l ]jbe 
in touch as soon as we can. 


Report on the 


STAT 


peeting 

as head of the 


of the Plans Staff of the DDO, is concerned with a worldwide picture of 


judging the performance of stations on the performance they have turned in 
against the objectives that have been set. This ties in with MBO. John has 

seen the | |series . Says DDO will proceed with MBO system; he is looking 

for something that can be applied to some of the jobs that they are doing in 

this evaluation work. told him this would be a requirement for a MBO 

packet of great interest in the Agency. We must get 2-3 days of MBO to 
relate to the Agency system. However, agency system is not uniform and it is ha 


hard to come up with a Agency-wide MBO packet. We have agreed to send 
all of the material that we had available on MBO for his use and for his staff's 

US * Approvetfor^efe h a ft se[2mi2zJ S/flffiMoflfoPP-O 18 < l uite 


simple; the problem is in the mechanics. 
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STATINTL 


STATINTL 


considered new. Critiques should be continued but in the limit that I am 
suggesting. What our front office needs is less paper, not more. Since 
course reports are directed to the DTR you would assume that their attach- 
ments are going to be read by him. But I would say that the critiques 
retained 

should be written by the unit involved and held on call. With exception of 
a new course... e.g., AMP where critiques are extremely important. I think 
a course report in its body should contain a summary of the salient comments 
made by the students, give a general flavor of how students reacted to the 


course. 

A routing for course reports: 

1. should go from the instructor to the branch or unit chief 

c/PDS/DTG (w/ critiques attached).. 


may rem<jy£ATINTL 


2 . then to 

critiques . . . 

3. Then to c/PDS 

4. Then to DTR 

Do I get reports on Ops courses? ....mainly those in FTD and the | | STATINTL 

courses... 

If there is anything of interest to SA/0Tf of course. Dale will send his 
reports to SA/OT, and then to I 


STATINTL 


There should be some formal policy statement. 

Each unit chief ought to devise a type of critique form which is good 
for his courses. 

I think evaluations, and we call them evaluations — not critiques, are 
very important and I think if you do it right you can get some useful feed- 
back. It depends on the course and the level of the people. Think the 
feedback is useful for course design and it's useful in terms of preparing 
a final report. Think it can be used in evaluating the course objectives. 
Think it's a criaiminal act to send forward handwritten critiques. 

Should consolidate all the answers to a certain question, but I am against 
the idea of summarizing and extracting favorable comments. 

In my experience, I read a lot of criti^ttS^-- for a year and gave back 
extensive remarks to instructors... and I sent back memoranda to the chief 
instructors. . .changes in courses resulted. . .e. ,g, China Ops, B0C...the 
iM&swpusi useful. 
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BTATINTL 


We have indicated to | ~| that we will do what we can. STATINTL 

At the training officers conference I was asked about an orientation for sub 
professionals (not clerical.} not professionals ) . . . I want to go to the senior 
training officers and get from them an idea of the population for such a 
course, their backlog, how many eod's annually in this category, and how 
seriours a matter this is to the sr training officers . 

Will get a bfcg response from NPIC. 

IWA is for newly eod’s professionals. 

Are we guilty of an elitist approach? 

The question is in an exploratory stage now. I would see this as a 3 day 
course to be done with my existing staff 
I asked 


STATINTL 

Ito tell us about additional day for DDO in next running 


STAT INTL 


of CIA Today and Tomorrow. 

This will be the fourth day starting with the January running, 
and I put our heads together and got a program which will be supplementary 
to the course. Picked subjects and people. We compromised on the question 
of OTR running it. We are going ahead this time in terms of reserving a 
room, scope notes, etc., but that after that Jack and Company will take it 
over. I posed the problem that if this IR^is successful we can't run 4th 
days for every outfit. But we are in good shape and will do it next time. 

STATINTL 

Will probably not put on it* because heA handling the fchs three 

day problem, but probably since they are $IA TINTL 

the DDO. 

TO THE AGENDA -..finally... 

First item: Course reports, critiques, and schedule formats: 

This is precisely where I came in. Found everything I read to be very famil- 
iar and really a bit boring. Would like to suggest that we not get into a 

as 

long historical discussion. I feel that course critiques/written by 
| students tell you nothing that you dont know in establishing courses and 
they don't provide you with anything useful. They are old hat and sometimes 
troublesome. They do provide a chance for the student to express himself, 
however. I would vote for continuing course critiques but I do not feel that 
| critiques should be sent further than the unit chief except for new 
courses . And after 3 runnings , in my book a course is no longer to be 
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STA TINTL 

j We would continue course critiques,, course report contain some summary of 
them; want to stop the paper tonnage from hitting the how. 

But don’t assume that what you get from the chief instructors as a summary of 
critiques is indeed what the critiques say. 

! There are lots of ways of critiquing a course. Take the BOC. Every 2-3 

j 

weeks, the chief instructor ax has a free-for-all with the students. They can 
Say anything they want, this is taken into account in the course report, 
but this gives you immediate and constant feedback. Yes, independent 
judgement is good if those making the Judgement know what isjlnvolved and know 
the details. 

There is a kind of feedback that is of use to the staff and there is also a 
different kind of report to someone independent. 

Concept of not needing critiques in long established courses if fallacious. 

This is based on the thought that courses never change. 

There are disparate courses and disparate views. Do you want uniformity 
or not; only way to getjit is to issue 0TR policy. 

Place no great stock in uniformity. It stifles and sometimes requires a 
lfittle dishonesty. 

I see no real point of disagreement here. 

Really the best way to do it is to have a guy who you are relying on to judge 

| your course, who designs a critiques to fit a course, and perhaps it should be 

! 

done on a selective basis. 

I 

Every student should have an opportunity to be heard. 

i 

! I think we should be gife&ing some sort of reading if we are to do the job we me 
set out to do in terms of objectives. 

: We use a simple matrix system which is quite helpful. 

' Quantititive approach tells you where the problem is; but doesn't tell you 
what the problem is. 

I think we should know whether or not we are hitting our objectives; we should 
add a paragraph in the critiques on this. 

This should be one of the meatiest parts of the course report. 

I think: we agree that critiques are expected as a general rule, format 

may vary, for all OTR courses offerings with minimum exceptions, 
Approved on whet her 
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STATINTL 


ILLEG B 


the course achieved its stated objective or objectives. 

Things someone n should try and their hand at the simple OTR policy statement 
on critiques. Need some kind of very basic critique guides . 

There is a difference in what is needed at the instructor level and at the 
DTR level. 

Critiques are required in every course, students are afforded an opportunity 
to write a critique in every course. Signing the critiques is optional. 

| Wants to see to it that the jni^i paper flow is reduced. If we don't attache 
the critiques, the DTR can always exercise his option to read and call for 
the critiques. 

j Will | | comment on the reports that go forward? STATINTL 

Right now, no. 

8k We should have a list of things in the policy paper which should be 
covered in critiques. But let's be flexible in format, 
j Policy has got to be broad. 

| there is no prohibitmtion in | * ' ‘ * [ routing a dro P||_|_EGIB 

! copy to anyone else affected by a part of the couree report? 

i 

j Right. 

I think it's unconceivable that we don't have a policy on critiques. Let's 

j get a policy with the latitude that Don is suggesting but containing funda- 

{ 

mental principles. 

A critique is basically an in-house document for the unit responsible for 
] that course. 

We/lll work on a policy statement, but the method of routing is decided. 


i COURSE REPORTS 


Should EE0 data be in course reports 

(mfe to see what statistics the registrar is compilong on their raciality 
and sexuality. . .and if there is still a requirements to do so.) 

Data base can easily be built in end of course reports. Minimum data is the 
class roster which is attached. Roster comes along with the course report. 

The information which tells you whether the right people are being sent to your 
course should be sent forward also. 

Issue is what OTR needs in order to have a data base. 


Approved For Rele 
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STATINTL 



reflect , Reommendations of individual unit chiefs should be determined 
here* 

Suggest that girls in registrafc office have a 5 x 8 card which checks pre- 
requisites. If student doesn't have, or if 73 is incomplete, send back for 
proper completion. 

to ask Elmer what statistics that he he feels he needs to have. 

ask elaer what do they report out on the EEO thing and from where does he 

get his data in order to make his report... 

A statistical sheet on the kind of people in the course is valuabble in 
some courses. 

Include directorate breakdowns? 
yes. 


ILLEGIB 


j Discussion of agenda for 7-8 November: 

1, Jontinuation of discussion on crituqies, end of course 

reports, schedule formats, etc. 

I 

! 2. requirements and production of TV tapes, 
drop analyst paper 
(check schedule to last minutes) 


i 
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